How about using pyprocessing to bypass the gil for autoparalleization to create an openmp like feature? It was accepted into python 2.7, so it will soon be a standard feature.
On Sat, Jun 21, 2008 at 10:24 AM, Stefan Behnel <[EMAIL PROTECTED]> wrote: > Hi, > > rahul garg wrote: >> I am working on a proposal to add some kind of a mechanism for a Python >> programmer to specify that a loop should be parallel. > > Sounds like a nice feature to me, although the GIL may make this more > expensive than you might think. > > >> The annotations will be >> added to only certain predefined type of loops since parallel iteration over >> generic objects is not possible. > > It's not necessarily impossible. Take one thread that iterates over the > iterable and hands each item to a somewhat long-running worker thread. > > >> I do not know if Cython has such a facility. > > No. > > >> My original suggestion : >> >> "pragma parallel for" >> for i in xrange(n):x[i] = y[i] > > I like the fact that it uses a string instead of a comment. However, I would > prefer having a real syntax in Cython. > > >> with parallel(i): >> for i in xrange(n):x[i] = y[i] > > Since we already have "with (no)gil", this would match nicely. Still, a syntax > inside the loop header would be preferable to keep the link with the actual > thing that runs in parallel, which is just the loop and nothing else inside > the "with" block. > > >> I think we can also do something like : >> >> for i in prange(i): x[i] = y[i] >> >> prange can simply default to xrange when running on the interpreter but the >> compiler can think of prange as a special object. > > But that would mean you'd have to change the code or at least provide the name > "prange" to make it run in Python, in which case there is less gain in keeping > up a Python-compatible syntax. > > >> Semantics needs to be defined properly. For example, exceptions cannot be >> raised in proper order in parallel loops. > > Order doesn't matter in non-synchronised parallel execution, but I would > expect them to be thread-local anyway. If you want to catch an exception > raised in a parallel loop, I'm fine with doing so inside the loop and loosing > the exception when the loop ends. > > >> Any ideas/suggestions/flames? OpenMP has many more features such as >> reduction variables. OpenMP is a very simple parallel programming model and >> introducing something similar in Python can be really helpful. >> >> For an implementation perspectives, it will be a little challenging to >> generate code while avoiding the GIL but it can be done in simple cases. > > It may even be enough to require "nogil" for parallel execution in the > beginning. I expect the GIL overhead to be too high for simple loops and I > don't think you'd write bigger things as a loop. When I use threads in Python, > I'm usually quite considerate about the minimum amount of work that gets > parallelised. > > Stefan > > _______________________________________________ > Cython-dev mailing list > [email protected] > http://codespeak.net/mailman/listinfo/cython-dev > _______________________________________________ Cython-dev mailing list [email protected] http://codespeak.net/mailman/listinfo/cython-dev
