I've developed an adverb for working with large files: http://code.jsoftware.com/wiki/NYCJUG/2014-05-13#Streaming_Through_Large_Files . An example of using this code can be found here - http://code.jsoftware.com/wiki/User:Devon_McCormick/Code/largeFileVet - and the updated version of the code here - http://code.jsoftware.com/wiki/User:Devon_McCormick/Code/WorkOnLargeFiles .
A good place to start might be with an example of using a simple version of an adverb that makes minimal assumptions about the logical structure of the file: http://code.jsoftware.com/wiki/User:Devon_McCormick/Code/WorkOnLargeFiles/SimpleFile . The complete code is here: http://code.jsoftware.com/wiki/User:Devon_McCormick/Code/workOnLargeFile.ijs . On Fri, Dec 25, 2015 at 7:37 AM, Raul Miller <[email protected]> wrote: > When you want to see how J will proceed, you can set up an experiment, > and use echo to show what is happening when. > > That said, your "0 verb will operate a box at a time (or a pair of > boxes at a time, since it's dyadic - the "0/ verb thus operating a > pair of boxes at a time but being monadic...). > > So... you'll be reading in a pair of files at a time, and accumulating > the results of your myverb in your J session. > > I hope this helps, > > -- > Raul > > > On Fri, Dec 25, 2015 at 4:51 AM, Ryan Eckbo <[email protected]> wrote: > > I'm processing some big files on the order of 2G, extracting >= 250M of > data > > from > > each. I have to memory map them to get the data: > > > > readbigfile=: 3 : 0 > > JCHAR map_jmf_ 'f';y > > NB. get data from f > > unmap_jmf_'f' > > ) > > > > I have about 150 of these files together with matching smaller ones, and > I > > need to > > do something like this: > > > > (fread@[ myverb readbigfile@])"0/ SmallFiles,.Bigfiles > > > > My question is how is the J runtime going to execute this: is it going to > > proceed line > > by line or try and read all the big files at once? If the former, is the > > memory freed > > right after execution? In general I don't know how to deal with huge > arrays. > > > > Thanks for any help, > > Ryan > > > > > > > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > -- Devon McCormick, CFA Quantitative Consultant ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
