DBMSs is a good point. A good counter-argument to 64-bit and hardware claim is the Meteor problem.
No need to be stigmatized over x5 memory and boxed results: a little imagination can help combine different techniques to achieve a solution. One such technique could be memory-mapped files. There is high chance that a mapped noun can be passed efficiently to regex. And as it was shown, regex over csv returns indices of rows, which can be stored and mapped themselves and accessed randomly. Thus you get efficient random access to rows of a csv file with very small memory footprint. It could be further extended to fields of rows. You could see an interactive demo of mapped files in media/wav addon. It zips fast through huge files (tested on ~100Mb, but could be more) with little physical memory consumed. --- Jack Andrews <[EMAIL PROTECTED]> wrote: > hi richard, hope this finds you well. > > commercial, terabyte dbs are not unusual -- hell, you can > buy a terabyte of disk pretty cheap. 64bit machines with > terabyte RAM are not. > > > Its hard to imagine a *real* file that cannot be handled > > in a 64bit machine. > > of course, any real file on a 64 bit machine can be read > with readcsv, as long as you've got RAM 5x the size of > the file. > > i've been given some alternatives to readcsv, but all of > them produce a huge boxed table for which i'll need > RAM proportional to the size of the file. > > i'm easily distracted by problems that need a more > elegant solution, and hence my ramblings earlier. my > recent posts may be more appropriate to jchat or, not > at all... > > > >From the earliest days in our industry wise > > heads have said its not worth yr time programming > > around hw limits. > > that DBMSes exist is a counter argument. > > ta, jack > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > ____________________________________________________________________________________ Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
