On Tue, Nov 4, 2008 at 5:31 AM, Leonardo Santagada <[EMAIL PROTECTED]> wrote: > > On Nov 4, 2008, at 12:11 AM, Josiah Carlson wrote: > >> There is no shortage of algorithms (such as matrix multiplication) that >> are parallelizable but not particularly good candidates for an IPC-based >> multiprocessing paradigm. >> >> Ahh, but those algorithms aren't going to be written in Python; they are >> going to be written in C, and are going to manipulate memory directly. With >> such things, you can use standard Python threads, call into your C runtime, >> and release the GIL. Alternatively, you can use the mmap module to store >> your data, shared across multiple processes, using the same direct-memory >> access as you would with multiple threads and GIL release. > > > That's the thing, right now you can't even use python to sketch one of those > algorithms and for example compare performance with a non parallel version.
Why not? The data needs to come from *somewhere*, it doesn't appear from the aether into your RAM. And even if it *does* appear from nowhere, you can always use synthetic data to compare single process vs. multiprocess - in pure Python. > Also with four cores on common intel processor you might be happy with your > algo being 4x faster while being quite easy to develop (not needing to use > c). Indeed, but it's not uncommon that using psyco you can get 2-4x (if not occasionally 10-100x), with similar (though generally superior) performance gains from using Pyrex/Cython. And really, after implementing your base algorithm in Python, translating it to C/C++ is generally pretty straightforward for anything not involving complex data structures or string parsing. > I personally think this will be too much work on And that's up to every programmer to decide for themselves. I've done a few of the things I've mentioned (shared mmaps, C code controlled by Python, etc.), and have been very pleased with the speedups I attained. With better tools (Python, multiprocessing, Pyrex/Cython, ...) now available, I can't help but say, "back in my day, we just wrote it in C and were *happy* that we didn't have to use Fortran". ;) - Josiah _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
