Hi Sean, On Sun, Jul 7, 2013 at 10:23 PM, Sean Fisk <seanf...@gmail.com> wrote: > Sorry, I didn't make it clear: my project is actually an interpreter for a > stencil-based language that is used for solving partial differential > equations. So I think that RPython is the right tool to an extent. It worked > very well for writing the interpreter (I also used Alex Gaynor's rply). > However, now I would like to parallelize the solver (i.e., the matrix > operations). > > I sounds as if this is where RPython stops being the right tool for the job, > since threading is useful only for concurrency and not speed, as in Python. > Thank you for letting me know before I spent too much time on it! I suppose > I should have guessed from the code I was writing.
Sorry for not answering you earlier. The point is that, indeed, you are right and I have no obvious solution to propose. RPython is really designed to write high-level interpreters in, and so it doesn't worry about real multicore usage. (In fact, the STM approach currently worked on is about giving multicore usage anyway, but without changing the basic model of a GIL, which is not what you need.) It all depends on what the performance characteristics of your interpreter are. Do you spend a lot of time in the interpreter itself, or rather in library code? In the latter case, RPython is not the best approach --- e.g. its JIT is useless, and it's a case where multithreading would really help. I'm guessing that your interpreter is somewhere in the middle... > I am seeing two options, on which I welcome comment: > > Continue using RPython and use rffi to make calls to C, where I implement > the solver function for high speed, optionally with threading or > message-passing. I've used pthreads, OpenMP, and MPI a number of times > before, but combining that with rffi might be a little over my head. > > Switch to using PyPy with multiprocessing and/or numpypy. Happily, my code > runs under PyPy and CPython albeit with some RPython libraries and some > RPython-specific workarounds (e.g., boxes). If you want to run several regular Python threads and still get multicore usage, one of the reasonably easy approaches right now would be to write pure Python code containing calls to C functions written with CFFI's verify(). These calls are done with the GIL released (both on CPython and PyPy). If instead you'd rather have a single Python thread driving calls to C functions that internally spawn several threads, then indeed OpenMP or MPI seems better. In this case it's easier to embed into an RPython program with rffi. You basically have to write C code that exposes some simple single-threaded API to the RPython program. You would use threads only internally, on the C side. A bientôt, Armin. _______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
