Re: Recommended number of threads? (in CPython)

2009-11-02 Thread Aahz
In article mailman.2256.1256831821.2807.python-l...@python.org, mk mrk...@gmail.com wrote: I wrote run-of-the-mill program for concurrent execution of ssh command over a large number of hosts. (someone may ask why reinvent the wheel when there's pssh and shmux around -- I'm not happy with

Re: Recommended number of threads? (in CPython)

2009-10-29 Thread Falcolas
On Oct 29, 9:56 am, mk mrk...@gmail.com wrote: Hello everyone, I wrote run-of-the-mill program for concurrent execution of ssh command over a large number of hosts. (someone may ask why reinvent the wheel when there's pssh and shmux around -- I'm not happy with working details and lack of

Re: Recommended number of threads? (in CPython)

2009-10-29 Thread Neil Hodgson
mk: I found that when using more than several hundred threads causes weird exceptions to be thrown *sometimes* (rarely actually, but it happens from time to time). If you are running on a 32-bit environment, it is common to run out of address space with many threads. Each thread allocates

Re: Recommended number of threads? (in CPython)

2009-10-29 Thread Paul Rubin
Neil Hodgson nyamatongwe+thun...@gmail.com writes: If you are running on a 32-bit environment, it is common to run out of address space with many threads. Each thread allocates a stack and this allocation may be as large as 10 Megabytes on Linux. I'm sure it's smaller than that under most

Re: Recommended number of threads? (in CPython)

2009-10-29 Thread Dave Angel
Paul Rubin wrote: Neil Hodgson nyamatongwe+thun...@gmail.com writes: If you are running on a 32-bit environment, it is common to run out of address space with many threads. Each thread allocates a stack and this allocation may be as large as 10 Megabytes on Linux. I'm sure it's