Actually, the bottleneck is not the CPU. Using Perfmon, I have verified that I can currently have 1000 threads running with under 25% CPU usage. The threads are fairly light weight. However, the stack size for each thread is by default 1MB, so the memory is the bottleneck. I want to have about 1000 threads, however, this requires that every machine I run my app on must have at least 1GB of memory. Not all my machines have this much memory.
I have read that you can change the default by recompiling Python and have at least 16 thousand threads. In fact, this is what I read at http://www.gossamer-threads.com/lists/python/python/354638: The only way to change the stack size is to define THREAD_STACK_SIZE in the thread_pthread.h header. There is an ifdef for the Mac that sets this to 64k. Just as a test I forced THREAD_STACK_SIZE to that value and recompiled Python. I can now create 16k threads although I'm not sure yet whether our application will run; some tuning is probably in order. I'm going to get the 2.2 source code and see what is different between the two and maybe get an idea of what a reasonable stack size is. However, this pertained to Linux and I am running my app Windows and I was looking for a solution that does not require a Python recompile. -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Tim Roberts Sent: Thursday, June 02, 2005 11:32 AM To: python-win32@python.org Subject: [python-win32] 1MB Thread Stack Size On Wed, 01 Jun 2005 17:25:50 -0700, "Hughes, Chad O" <[EMAIL PROTECTED]> wrote: >I have a program that needs to create a great deal of threads. >Unfortunately, I cannot seem to find a way to lower the 1MB default >stack size per thread. The threading module does not seem to support >setting the stack size explicitly. I have 1GB of memory on my system >so I can only create about 1000 threads before I receive the following >error trace: > > It is almost impossible to imagine a situation in which 1,000 threads actually provides any productivity. Any time the number of threads in a process is more than an order of magnitude greater than the number of CPUs, the operating system administration overhead begins to overwhelm the productive work that can be done. Surely you could solve your problem with 100 threads and some appropriate queueing? -- Tim Roberts, [EMAIL PROTECTED] Providenza & Boekelheide, Inc. _______________________________________________ Python-win32 mailing list Python-win32@python.org http://mail.python.org/mailman/listinfo/python-win32 _______________________________________________ Python-win32 mailing list Python-win32@python.org http://mail.python.org/mailman/listinfo/python-win32
