I've been playing with medusa and it sometimes does what I expect, but sometimes it's doing something strange and I'm wondering if someone can help me to do one specific medusa-like thing but more simply (and without the strangeness, which I haven't fully traced but I hope to avoid having to).
What I want is a version of pmap that will always use any available cores to compute remaining values (except of course the last couple of values, when there are less remaining values than cores). In other words, I want the behavior that Andy Fingerhut describes medusa as having here: > On Sep 22, 2011, at 11:34 PM, Andy Fingerhut wrote: > > > pmap will limit the maximum number of simultaneous threads. So will the > > medusa library's medusa-pmap. > > > > The difference is that if one job early in the list takes, e.g., 100 times > > longer than the next 20, and you have 4 cpus available, pmap will start the > > first (4+2)=6 in parallel threads, let jobs 2 through 6 complete, and then > > wait for the first one to finish before starting number 7. Thus most of > > the time will be spent running one thread. This has the advantage of > > limiting the memory required to store intermediate results, but the > > disadvantage of low usage of multiple cpu cores. > > > > Medusa has a medusa-pmap that will also limit the parallelism, but it lets > > you pick the level of parallelism (it isn't fixed at (# cpus + 2)), and it > > will continue starting new threads, even if that means using more memory > > when one job takes much longer than the following jobs. > > FWIW I'll always be calling this on a finite sequence (although it may have 10000 elements or so, so I presume that I shouldn't start threads for all of them at once), and I will want a fully realized result sequence so I'll surround calls with doall if necessary. I will want all of the pmapped computations to finish before I continue doing anything else. I know that I could launch the threads in other ways -- e.g. I sometimes use agents for something similar -- but pmap is much more elegant in many cases, and anyway I'm not even sure I'm getting full utilization with my other methods. The medusa-pmap function is very close to what I want but medusa seems to do more that what I need, it requires initi/startup calls, it involves timeouts which I will never want, and it behaves strangely when I run my code on a 48 core node. (It runs fine on my 4-core laptop, and then it seems to work beautifully on the 48 core node too for a while, giving me nearly 4800% utilization, but then it does something that looks like it might be caused by everything hitting the timeouts... which I bumped up by several orders of magnitude but I'm still getting the weird behavior). So: Is there a simpler way to get "pmap but not lazy and use all cores fully until the whole result sequence is completed"? This is usually what I really want when I'm writing code that utilizes concurrency. I've looked at the pmap source but it isn't obvious to me how to change that to do what I want... So any pointers would be appreciated. Thanks, -Lee -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
