On Dec 13, 2012, at 4:21 PM, cameron wrote: > > Have you made any progress on a small deterministic benchmark that reflects > your applications behaviour (ie. the RNG seed work you were discussing)? I'm > keen to help, but I don't have time to look at benchmarks that take hours to > run. > > I've also been using clojure for genetic programming and have been getting > very good parallel speedups on a single jvm, this type of problem should see > excellent performance gains, it has many medium size cpu intensive > sub-problems that require no shared resources. I used hadoop to scale beyond > single machines, in the worst case you can use this approach and you should > see speed-ups equivalent to your multiple jvm test. I just split the > population into small blocks and had each map function calculate the fitness > and return a map of individual-id -> fitness. > > Cameron.
I don't think we've quite worked out a deterministic test, but I'll look into it some more. I'd be interested in seeing your GP system. The one we're using evolves "Push" programs and I suspect that whatever's triggering this problem with multicore utilization is stemming from something in the inner loop of my Push interpreter (https://github.com/lspector/Clojush)... but I don't know exactly what it is. I've never used hadoop, but I've done lots of coarse-grained parallel GP using (gasp) shell scripts and the file system to glue the independent parts together. I realize that I could go back to this sort of thing, or something more modern and reasonable like hadoop, but of course it'd be a lot nicer if parallelization via agents (or some other mechanism central to the language) just didn't have whatever pathology we've uncovered. -Lee -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
