Josiah Carlson wrote:
On Mon, Nov 3, 2008 at 10:59 AM, Curt Hagenlocher
<[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> wrote:
On Mon, Nov 3, 2008 at 11:50 AM, Josiah Carlson
<[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> wrote:
On Sun, Nov 2, 2008 at 3:51 PM, <[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>> wrote:
Antoine> I think it is important to remind that the GIL
doesn't prevent
Antoine> (almost) true multithreading. The only thing
it prevents is
Antoine> full use of multi-CPU resources in a single
process.
I believe everyone here knows that. I believe what most
people are
clamoring for is to make "full use of their multi-CPU
resources in a single
process".
Which is, arguably, silly. As we've seen in the last 2 months
with Chrome, multiple processes for a single "program" is
actually a pretty good idea. With the multiprocessing module
in the standard library offering a threading-like interface,
people no longer have any excuses for not fully exploiting
their multiple cores in Python.
There is no shortage of algorithms (such as matrix multiplication)
that are parallelizable but not particularly good candidates for
an IPC-based multiprocessing paradigm.
Ahh, but those algorithms aren't going to be written in Python; they
are going to be written in C, and are going to manipulate memory
directly. With such things, you can use standard Python threads, call
into your C runtime, and release the GIL. Alternatively, you can use
the mmap module to store your data, shared across multiple processes,
using the same direct-memory access as you would with multiple threads
and GIL release.
That's only one example of an algorithm where IPC isn't suitable. And
you're right, anyone with this kind of use case is just not going to use
Python. That isn't a good thing.
Michael Foord
Also, most local-only communications primitives (named pipes,
anonymous pipes, unix domain sockets, ...) use zero/one copy
implementations, so as long as your RPC isn't slow, you can do pretty
well even on the Python side (especially if you pre-allocate your
receive buffer, and fill in the data as you receive it; this is pretty
much what mutablebytes was created for, now we just need a proper
memoryview).
So again, I claim that not using multiple processes for your
multi-core machine in order to use your multiple cores to their full
extent is silly. As an aside, Python array.array() instances have a
char* and length, so if you are careful, you can create an array.array
object from an mmap pointer, and get fairly decent performance even in
Python with shared memory. Someone should probably look into allowing
array.array() to take a read/readwrite memoryview as an argument to
support such things, as well as allowing mmaps to be passed via
multiprocessing (if they aren't already supported).
- Josiah
------------------------------------------------------------------------
_______________________________________________
Python-Dev mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog
_______________________________________________
Users mailing list
Users@lists.ironpython.com
http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
