Output of readcsv is box array. How well is current jmf in supporting box array?
Oleg Kobchenko wrote:
DBMSs is a good point.
A good counter-argument to 64-bit and hardware
claim is the Meteor problem.
No need to be stigmatized over x5 memory and
boxed results: a little imagination can help combine
different techniques to achieve a solution.
One such technique could be memory-mapped files.
There is high chance that a mapped noun can be
passed efficiently to regex. And as it was shown,
regex over csv returns indices of rows, which
can be stored and mapped themselves and accessed
randomly. Thus you get efficient random access
to rows of a csv file with very small memory footprint.
It could be further extended to fields of rows.
You could see an interactive demo of mapped files
in media/wav addon. It zips fast through huge files
(tested on ~100Mb, but could be more) with
little physical memory consumed.
--- Jack Andrews <[EMAIL PROTECTED]> wrote:
hi richard, hope this finds you well.
commercial, terabyte dbs are not unusual -- hell, you can
buy a terabyte of disk pretty cheap. 64bit machines with
terabyte RAM are not.
Its hard to imagine a *real* file that cannot be handled
in a 64bit machine.
of course, any real file on a 64 bit machine can be read
with readcsv, as long as you've got RAM 5x the size of
the file.
i've been given some alternatives to readcsv, but all of
them produce a huge boxed table for which i'll need
RAM proportional to the size of the file.
i'm easily distracted by problems that need a more
elegant solution, and hence my ramblings earlier. my
recent posts may be more appropriate to jchat or, not
at all...
>From the earliest days in our industry wise
heads have said its not worth yr time programming
around hw limits.
that DBMSes exist is a counter argument.
ta, jack
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
