On 5/8/07, Shannon -jj Behrens <[EMAIL PROTECTED]> wrote:
On 5/8/07, Andrew Wilson <[EMAIL PROTECTED]> wrote: > It would be nice to have most of the serving done from one process. In case > you care, here's why: When running Python under Virtuzzo (with a Red Hat or > CentOS kernel), instantiating the smallest thread takes up 10mb of our > memory allocation. On a virtual machine running with 256mb of assigned > memory, this means that a 10-thread Python process burns up almost half our > total memory. We are trying to figure out why this happens, and how we can > avoid it, I've heard of this happening before. The trick is to tell your hosting company not to double count shared libraries.
Actually, after long hours of tracking and testing, I can definitely confirm that this is not the case. What's actually going on (for those who may or may not care) is this: under many flavors of Linux, the standard pthread library allocates 8MB of stack space for each new thread. This isn't typically a problem, because Linux apparently adopts a (to me, absurd) philosophy of "over-commit" combined with a kernel that uses "defered allocation". Upshot: the "virtual" memory goes up by 8MB+ for each new pthread, but the actually used memory only goes up by a small amount. By specifying a smaller stack size (512KB, for instance), we can spawn a reasonably high number of threads within our virtual machine's bean-counter limits. Happily, Python 2.5 adds a "stack_size()" function to the threading modules for just this purpose. Unhappily, it doesn't work as advertised; when called with no arguments, the function resets the stack size to the default stack size, instead of returning the current stack size. This is, to me, a consequence of taking a perfectly reasonable "get/set()" API and trying to squash it into one function. -- Andrew
_______________________________________________ Paste-users mailing list [email protected] http://webwareforpython.org/cgi-bin/mailman/listinfo/paste-users
