On Tue, Jul 30, 2013 at 3:38 AM, Armin Rigo <ar...@tunes.org> wrote: > Hi Sean, > > On Sun, Jul 7, 2013 at 10:23 PM, Sean Fisk <seanf...@gmail.com> wrote: > > Sorry, I didn't make it clear: my project is actually an interpreter for > a > > stencil-based language that is used for solving partial differential > > equations. So I think that RPython is the right tool to an extent. It > worked > > very well for writing the interpreter (I also used Alex Gaynor's rply). > > However, now I would like to parallelize the solver (i.e., the matrix > > operations). > > > > I sounds as if this is where RPython stops being the right tool for the > job, > > since threading is useful only for concurrency and not speed, as in > Python. > > Thank you for letting me know before I spent too much time on it! I > suppose > > I should have guessed from the code I was writing. > > Sorry for not answering you earlier. The point is that, indeed, you > are right and I have no obvious solution to propose. RPython is > really designed to write high-level interpreters in, and so it doesn't > worry about real multicore usage. (In fact, the STM approach > currently worked on is about giving multicore usage anyway, but > without changing the basic model of a GIL, which is not what you > need.) > > It all depends on what the performance characteristics of your > interpreter are. Do you spend a lot of time in the interpreter > itself, or rather in library code? In the latter case, RPython is not > the best approach --- e.g. its JIT is useless, and it's a case where > multithreading would really help. I'm guessing that your interpreter > is somewhere in the middle... >
I am in the process of profiling and running benchmarks right now, but that is absolutely something I would need to find out before starting on any parallel solution. > > > I am seeing two options, on which I welcome comment: > > > > Continue using RPython and use rffi to make calls to C, where I implement > > the solver function for high speed, optionally with threading or > > message-passing. I've used pthreads, OpenMP, and MPI a number of times > > before, but combining that with rffi might be a little over my head. > > > > Switch to using PyPy with multiprocessing and/or numpypy. Happily, my > code > > runs under PyPy and CPython albeit with some RPython libraries and some > > RPython-specific workarounds (e.g., boxes). > > If you want to run several regular Python threads and still get > multicore usage, one of the reasonably easy approaches right now would > be to write pure Python code containing calls to C functions written > with CFFI's verify(). These calls are done with the GIL released > (both on CPython and PyPy). > > If instead you'd rather have a single Python thread driving calls to C > functions that internally spawn several threads, then indeed OpenMP or > MPI seems better. In this case it's easier to embed into an RPython > program with rffi. You basically have to write C code that exposes > some simple single-threaded API to the RPython program. You would use > threads only internally, on the C side. > These are both very reasonable suggestions. Thank you for clearly spelling out my best options. Unfortunately, my internship is coming to an end, and I am unsure whether I will get the chance to implement them. If I do implement them, however, I'll make a point to write about my experience so that others can benefit from these solutions. Thank you for the answer, Armin. - Sean > > > A bientôt, > > Armin. >
_______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
