I reran the tests with 8 threads on my machine and watched it in the built in GUI process manager on the Mac (Activity Monitor), and I see what you mean. The load is distributed evenly until near the end, when the processes drop one by one and only one is left. I don't think it matters how many cores you have to get this working, because the operating system will distribute the threads evenly anyway (though the cores to matter for speed).
It looks like there are two classes for use in IPython, the MultiEngine Client [0] and the Task Client [1]. The MultiEngine Client, which is what you are currently using, allows you to manage each engine, but requires you to do your own load balancing. The Task Client doesn't give you as much control, but does automatic load balancing. I will attempt to convert your file to the Task Client and see if that is any better. [0] - http://ipython.scipy.org/doc/stable/html/parallel/parallel_multiengine.html [1] - http://ipython.scipy.org/doc/stable/html/parallel/parallel_task.html Aaron Meurer On Jul 16, 2009, at 5:19 PM, Ondrej Certik wrote: > > On Thu, Jul 16, 2009 at 5:16 PM, Aaron S. Meurer<[email protected]> > wrote: >> >> Yep. It is definitely faster: >> >> $ time ./bin/test >> <testing> >> real 8m18.517s >> user 4m6.898s >> sys 0m4.019s >> $ time python t.py (with ipcluster local -n 2) >> <testing> >> real 5m31.959s >> user 0m0.602s >> sys 0m0.290s >> $ time python t.py (with ipcluster local -n 4) >> <testing> >> real 4m30.470s >> user 0m0.494s >> sys 0m0.227s >> $ time python t.py (with ipcluster local -n 8) >> <testing> >> real 6m0.355s >> user 0m0.654s >> sys 0m0.229s >> >> I only have 2 cores, but I think it benefits from having more threads >> anyway. I think I can see what you mean with the 8 threads >> though. I >> will need to find a process manager for the Mac that lets you view >> what different threads are doing and on what cores. htop is Linux >> only. > > If you want an access on 8 cores linux machine, send me your public > ssh key offlist. > > What happens there is that after about 10s, all cores are done and are > waiting about 30s for one core to finish. So we need some way to > automatically distribute the work on the nodes. Actually, maybe > ipython has some feature for this too (load balancing). > > Ondrej > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "sympy" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/sympy?hl=en -~----------~----~----~----~------~----~------~--~---
