Hi, the underlying parallel mechanism is processes communicating via execnet.
-- Ronny On 01/26/2012 12:30 AM, Ateljevich, Eli wrote: > Thanks. I really appreciate the help. I have one last question -- is the > underlying parallel mechanism threads or mpi or something else? Do I have to > worry about things like MPI communicators getting in each other's way? > > Cheers, > Eli > > -----Original Message----- > From: holger krekel [mailto:hol...@merlinux.eu] > Sent: Friday, January 20, 2012 2:27 PM > To: Ateljevich, Eli > Cc: holger krekel; py-dev@codespeak.net > Subject: Re: [py-dev] xdist and thread-safe resource counting > > On Fri, Jan 20, 2012 at 12:56 -0800, Ateljevich, Eli wrote: >> Thanks, Holger. I appreciate the hint about where to do the testing and >> waiting in pytest_runtest_setup and I think the atomic file rename idea is >> an interesting way to set up a signal. >> >> I may not fully understand about xdist. I certainly agree it is efficient >> use of mpirun that is the crux of doing the job right ... so probably any >> load balancing offered by xdist is going to be wasted on me. >> >> The main service I was looking for out of xdist was the ability to run tests >> concurrently. As I think you realize, if I have a pool of 16 processors and >> the first four tests collected require 8, 4, 8, 4 processors, I would want >> this behavior: >> 1. the first test to start immediately >> 2. the second test to start immediately without the first finishing >> 3. the third test to either wait or start in a python sense but "sleep" >> before launching mpi >> 4. the fourth test to start immediately >> >> Is vanilla py.test able to do this kind of concurrent testing? Or would I >> need to tweak it to launch tests in threads according to my criterion for >> readiness? > > A run with pytest-xdist, notably, "py.test -nNUM" allows to implement this > behaviour, i think. > >> I think we have settled how I would allocate resources, but your idea >> implies I might have all the test hints in one place. If I have full control >> all the test launches this might allow me to do some sort of knapsack >> problem-ish kind of reorganization to keep everything fully utilized rather >> than taking the test in the order they were collected. For instance, if I >> had 16 processors and the first four tests take 12-12-4-4 I could do this in >> the order (12+4 concurrently) (12+4 concurrently). Do I have this level of >> control? > > I think so yes. IIRC pytest-xdist distributed the first four tests > such that they each land at different nodes. So, given the algorithm > i hinted at, and running with "py.test -n3" the first sub process would > start and run on 12 processors. The second process would see that > there are 12 used and wait until 12 become available. The > third process would only need 4 and immediatly continue, utilizing > all 16 processors at that time. When the first one finishes the > second sub process would see that there now are enough and proceed > with its testing. This is all fully compatible with pytest-xdist > semantics and only needs code at pytest_runtest_setup time i think. > > best, > holger > > _______________________________________________ > py-dev mailing list > py-dev@codespeak.net > http://codespeak.net/mailman/listinfo/py-dev
signature.asc
Description: OpenPGP digital signature
_______________________________________________ py-dev mailing list py-dev@codespeak.net http://codespeak.net/mailman/listinfo/py-dev
