The out-of-memory is due to host OS page file limits (or perhaps user limits). It is related to virtual memory, not real memory. I'm not sure, but the mapped files lab may make this more clear.
The solution depends on changes in your host environment, not on J. If you fix this problem, you may just hit the problems that Raul points out. That is, if you are accessing too much more virtual memory than you have real memory, your system will start to thrash disk and run verrrry slowly. On Tue, May 8, 2012 at 6:56 AM, Joe Bohart <jboh...@gmail.com> wrote: > Hi J'ers, > > I'm trying to perform calculations on massive data sets using mapped files > and after searching the forum/essays am stuck. > > I've load 50 million integer digits of pi and trying to do a moving average > on them. I 've created two mapped files one for the data and one for the > calculated results, but I get out of memory errors. I'm not seeing what I'm > doing wrong here. > > My goal is to work with floating point numbers eventually, but i'd figure > i'd start with integers first, since there are more examples for integers > than floats in the 'mapped files' Labs (if anyone has the code for the > example mention in the 67 of 68 slide of the Lab on mapped files using NYSE > stock prices - I'd love to see it). > > load 'jmf files dir' > NB. data from http://zenwerx.com/projects/pi-digits/pi/ > NB. used perl to write each digits on 1 line of file data > pidata =: fread '/home/joe/pi-study/data' > NB. type is 2 (literal) > 3!:0 pidata > NB. convert to 4-byte integers > piInt =: _2 (3!:4) pidata > NB. shape is 50,000,000 > $piInt > ]sz=:7!:5 <'piInt' > NB. define moving average > ma =: mean \ > NB. out of memory ma, expected since storing results in ram > smoothPi =: 100 ma piInt > > NB. create jmf for results > fn =: jpath '~temp/pi-results.jmf' > ]sz=: 7!:5 <'piInt' > createjmf_jmf_ fn;sz > JFL map_jmf_ 'piResults'; fn > piResults =. 100 ma piInt > NB. still get out of memory! > > Much thanks ! > Joe > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
