That's an interesting direction of thought too. There are two kinds of mapped nouns: dyad is for a list of given cell shapes (can't be boxed) and monad is for single J noun in the whole file (where boxed can be used).
So the monad case could be used for a boxed table, which would be populated by appending rows. The there are few caveats: - the readcsv equivalent need to know that it outputs to a mapped noun - there need to be a mechanism to automatically grow the file as the table grows But once such boxed mapped file is created it can be mapped and operated with random access or as any other boxed table with very low memory footprint. --- bill lam <[EMAIL PROTECTED]> wrote: > Output of readcsv is box array. How well is current jmf in supporting box > array? > > Oleg Kobchenko wrote: > > DBMSs is a good point. > > > > A good counter-argument to 64-bit and hardware > > claim is the Meteor problem. > > > > No need to be stigmatized over x5 memory and > > boxed results: a little imagination can help combine > > different techniques to achieve a solution. > > > > One such technique could be memory-mapped files. > > There is high chance that a mapped noun can be > > passed efficiently to regex. And as it was shown, > > regex over csv returns indices of rows, which > > can be stored and mapped themselves and accessed > > randomly. Thus you get efficient random access > > to rows of a csv file with very small memory footprint. > > It could be further extended to fields of rows. > > > > You could see an interactive demo of mapped files > > in media/wav addon. It zips fast through huge files > > (tested on ~100Mb, but could be more) with > > little physical memory consumed. > > > > > > --- Jack Andrews <[EMAIL PROTECTED]> wrote: > > > >> hi richard, hope this finds you well. > >> > >> commercial, terabyte dbs are not unusual -- hell, you can > >> buy a terabyte of disk pretty cheap. 64bit machines with > >> terabyte RAM are not. > >> > >>> Its hard to imagine a *real* file that cannot be handled > >>> in a 64bit machine. > >> of course, any real file on a 64 bit machine can be read > >> with readcsv, as long as you've got RAM 5x the size of > >> the file. > >> > >> i've been given some alternatives to readcsv, but all of > >> them produce a huge boxed table for which i'll need > >> RAM proportional to the size of the file. > >> > >> i'm easily distracted by problems that need a more > >> elegant solution, and hence my ramblings earlier. my > >> recent posts may be more appropriate to jchat or, not > >> at all... > >> > >>> >From the earliest days in our industry wise > >>> heads have said its not worth yr time programming > >>> around hw limits. > >> that DBMSes exist is a counter argument. > >> > >> ta, jack > >> ---------------------------------------------------------------------- > >> For information about J forums see http://www.jsoftware.com/forums.htm > >> > > > > > > > > > > ____________________________________________________________________________________ > > Be a better friend, newshound, and > > know-it-all with Yahoo! Mobile. Try it now. > http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ > > > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > > > > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > ____________________________________________________________________________________ Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
