If you want "good performance" you'll need to do a little thinking about where your resources are getting eaten. (Which also means some measurement and testing.)
Specifically, it can really help to make use of available memory instead of relying entirely on loops. In other words, if you are running out of memory, run some timings on the problem with the memory intensive solution and the looping solution. If your memory intensive solution is significantly faster, do not entirely abandon it - instead, use it on digestible sized blocks and accumulate those results using a loop, In general terms, it can help to think of a memory vs. time graph which looks something like this: require'plot' plot (% >:@i.)100 ... and remembering that you are trying to minimize time consumed but you are hitting a ceiling on available memory. And, yes, the resulting implementation can look a bit ugly sometimes - it can look like a violation of "don't repeat yourself". But all abstractions leak. (If you want to take this further, you would need to also have knowledge of and tune for the cache sizes on your machine. But that's probably not going to be useful for this specific exercise.) I hope this helps. Thanks, -- Raul On Tue, Feb 21, 2017 at 8:41 AM, Paul Moore <[email protected]> wrote: > On 21 February 2017 at 13:23, Joe Bogner <[email protected]> wrote: >>> >>> The key thing here is that I'm going to be collecting millions of >>> results, and I'll run out of memory if I just retain them all and >>> count at the end. But I'm not sure how to modify structures in J. >>> >> >> Are you sure you will run out of memory? I can hold a billion ints on my >> machine and use 7.8gb of memory. > > I did yesterday. I can't recall the details - I may have been running > a billion simulations. I may well also have been using a > memory-inefficient approach, though (if I held 5 numbers for each run, > that would be 40GB of RAM I guess). > > My purpose behind this exercise is that I'm trying to demonstrate to a > friend that you don't need to write custom C programs (which is what > he did) to do this type of work, and that off the shelf (albeit > specialised) languages can cope fine (where "fine" means "comparable > to custom C"). So far, I'm finding that's not true - my experiments > with various languages[1] have either been too slow or too memory > hungry to make my case. When a tiny performance issue or over-use of > memory in the implementation of the single experiment gets multiplied > by factors on the order of millions or billions, that doesn't scale > well enough. > > It's an interesting exercise that's got me learning a lot about > various languages' strengths and weaknesses, though :-) > > Paul > > [1] Most of which have been at the "interested beginner" level, as here. > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
