Oops, it looks like Mike Day already covered this topic (with more specifics than I used).
-- Raul On Tue, Feb 21, 2017 at 10:42 AM, Raul Miller <[email protected]> wrote: > If you want "good performance" you'll need to do a little thinking > about where your resources are getting eaten. (Which also means some > measurement and testing.) > > Specifically, it can really help to make use of available memory > instead of relying entirely on loops. > > In other words, if you are running out of memory, run some timings on > the problem with the memory intensive solution and the looping > solution. If your memory intensive solution is significantly faster, > do not entirely abandon it - instead, use it on digestible sized > blocks and accumulate those results using a loop, > > In general terms, it can help to think of a memory vs. time graph > which looks something like this: > > require'plot' > plot (% >:@i.)100 > > ... and remembering that you are trying to minimize time consumed but > you are hitting a ceiling on available memory. > > And, yes, the resulting implementation can look a bit ugly sometimes - > it can look like a violation of "don't repeat yourself". But all > abstractions leak. > > (If you want to take this further, you would need to also have > knowledge of and tune for the cache sizes on your machine. But that's > probably not going to be useful for this specific exercise.) > > I hope this helps. > > Thanks, > > -- > Raul > > > On Tue, Feb 21, 2017 at 8:41 AM, Paul Moore <[email protected]> wrote: >> On 21 February 2017 at 13:23, Joe Bogner <[email protected]> wrote: >>>> >>>> The key thing here is that I'm going to be collecting millions of >>>> results, and I'll run out of memory if I just retain them all and >>>> count at the end. But I'm not sure how to modify structures in J. >>>> >>> >>> Are you sure you will run out of memory? I can hold a billion ints on my >>> machine and use 7.8gb of memory. >> >> I did yesterday. I can't recall the details - I may have been running >> a billion simulations. I may well also have been using a >> memory-inefficient approach, though (if I held 5 numbers for each run, >> that would be 40GB of RAM I guess). >> >> My purpose behind this exercise is that I'm trying to demonstrate to a >> friend that you don't need to write custom C programs (which is what >> he did) to do this type of work, and that off the shelf (albeit >> specialised) languages can cope fine (where "fine" means "comparable >> to custom C"). So far, I'm finding that's not true - my experiments >> with various languages[1] have either been too slow or too memory >> hungry to make my case. When a tiny performance issue or over-use of >> memory in the implementation of the single experiment gets multiplied >> by factors on the order of millions or billions, that doesn't scale >> well enough. >> >> It's an interesting exercise that's got me learning a lot about >> various languages' strengths and weaknesses, though :-) >> >> Paul >> >> [1] Most of which have been at the "interested beginner" level, as here. >> ---------------------------------------------------------------------- >> For information about J forums see http://www.jsoftware.com/forums.htm ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
