I'm processing some big files on the order of 2G, extracting >= 250M of
data from
each. I have to memory map them to get the data:
readbigfile=: 3 : 0
JCHAR map_jmf_ 'f';y
NB. get data from f
unmap_jmf_'f'
)
I have about 150 of these files together with matching smaller ones, and
I need to
do something like this:
(fread@[ myverb readbigfile@])"0/ SmallFiles,.Bigfiles
My question is how is the J runtime going to execute this: is it going
to proceed line
by line or try and read all the big files at once? If the former, is
the memory freed
right after execution? In general I don't know how to deal with huge
arrays.
Thanks for any help,
Ryan
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
