On Wed, May 9, 2012 at 7:02 AM, Joe Bohart <jboh...@gmail.com> wrote: > my goal is to have a practical example of running calculations on data that > is larger than my machines ram.
I used to do this a lot, back when machine ram sizes were much smaller. They key issue here, for my favored approach, is to break the problem up into blocks that can fit in ram. The details of how to best do this depends on your application. But, in essence, you want a block size large enough to mask the costs of the blocking algorithm and small enough to fit in memory. 10MB can be a nice block size, but you might vary that by an order of magnitude, or more, depending on your application. Alternatively, if you want a more general approach, you can use virtual ram and delegate the problem to your OS. (You will also probably want to delegate shutting J down when it's being ill behaved to your OS, if you go here.) > I'm thinking mapped files is the only way to do that and if I > understand..... I *hope* it's possible as long as I make sure the > intermediate results do not exceed J session memory (i.e. perhaps close to > machine ram). I have not tested whether we can map files which are larger than the allocation limit. I imagine we should be able to. That would get you a value that you have to extract blocks of data from before you use them. However, you can also use indexed file reads to accomplish this without mmap. See http://www.jsoftware.com/help/dictionary/dx001.htm Typically, I would expect mmap to perform better than indexed file read, but that's based on the OS implementations I have had experience with. -- Raul ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
