mk wrote:
sjdevn...@yahoo.com wrote:
On Feb 20, 9:58 pm, John Nagle <na...@animats.com> wrote:
sjdevn...@yahoo.com wrote:
On Feb 18, 2:58 pm, John Nagle <na...@animats.com> wrote:
Multiple processes are not the answer. That means loading
multiple
copies of the same code into different areas of memory. The cache
miss rate goes up accordingly.
A decent OS will use copy-on-write with forked processes, which should
carry through to the cache for the code.
That doesn't help much if you're using the subprocess module. The
C code of the interpreter is shared, but all the code generated from
Python is not.
Of course. Multithreading also fails miserably if the threads all try
to call exec() or the equivalent.
It works fine if you use os.fork().
What about just using subprocess module to run system commands in worker
threads? Is this likely to have problems?
Regards,
mk
The discussion above was about using "fork" to avoid duplicating the
entire Python environment for each subprocess. If you use the subprocess
module, you load a new program, so you don't get much memory sharing.
This increases the number of cache misses, which is a major bottleneck
in many-CPU systems with shared caches.
The issue being discussed was scaling Python for CPUs with many cores.
With Intel shipping 4 cores/8 hyperthread CPUs, the 6/12 part working,
and the 8/16 part coming along, this is more than a theoretical
issue.
John Nagle
--
http://mail.python.org/mailman/listinfo/python-list
