On Apr 24, 3:51 am, "Devin Torres" <[EMAIL PROTECTED]> wrote: > So we're using Pylons and Python in general for our new company > platform. We just bought a server with 4 cores to help us reach our > scalability goals, but there are a few questions I'm interested in > asking the Pylons community. > > I (mostly) understand the nature of "threads" in Python. From my > understanding, the GIL locks the interpreter to executing only one > Python thread at a time, but C modules can take advantage of a Python > application being multithreaded, because they can operate independant > of the GIL. Presumably, this would mean that there is, in fact, a > benefit to using threads in Paste, because most network I/O bound > stuff happens within a C module. > > Given this situation, I believe that despite paste making an effort to > be multithreaded, it would still be advantageous to run a cluster of > four Pylons instances and proxy to these using nginx. > > Using our setup we'd have four pylons instances being proxied to by > four nginx worker threads. > > In nginx you can set the processor affinity for each worker thread, > thus placing each worker on a different core 0..3. > > Here's where things get tricky: > I've found a Python package that apparently allows Python applications > to set their processor affinity (I'm afraid it doesn't work on OS > X):http://pypi.python.org/pypi/affinity/0.1.0 > > Using this, what do you guys thing on my idea to write a custom > cluster controller, perhaps using supervisord, that will start nginx > and the four worker processes, and then fork()'s my Pylons app into > into a cluster of four? > > Is this overkill? Is Paste more mulithreaded than I'm giving it credit > for? Is there a better way to go about this? Does an alternative to > the 'affinity' package exist?
Use Apache and mod_wsgi and you have all that you want except playing with 'processor affinity'. This is because Apache is multi process by design and thus can properly make use of multiple CPUs. A lot of what goes on in Apache is also not implemented in Python and thus not subject to GIL issues. You might also have a read of the following: http://blog.dscpl.com.au/2007/09/parallel-python-discussion-and-modwsgi.html http://blog.dscpl.com.au/2007/07/web-hosting-landscape-and-modwsgi.html These explain some of these issues about multiprocess web servers and the GIL. Not sure why you just wouldn't let the operating system handle allocation of processes/threads across CPUs as it is likely in general to do a better job. Are you sure you aren't trying to solve a problem that doesn't really exist. Graham --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "pylons-discuss" group. To post to this group, send email to pylons-discuss@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en -~----------~----~----~----~------~----~------~--~---
