On 8 April 2010 09:34, Michael Schurter <[email protected]> wrote: > On Wed, Apr 7, 2010 at 4:09 PM, Graham Dumpleton > <[email protected]> wrote: >> On 8 April 2010 07:43, Michael Schurter <[email protected]> wrote: >>> On Wed, Apr 7, 2010 at 2:02 PM, Michael Dirolf <[email protected]> wrote: >>>> As a follow-up, there is a fix that went into PyMongo since 1.5.2 for >>>> behavior similar to this. I think the OP is looking into this now, but >>>> unless the master of PyMongo fails (against any version of modwsgi), I >>>> think that was probably the issue... >>> >>> Sorry for the noise. It appears that it was simply a bug in PyMongo >>> 1.5.2 and a Python path error led me to believe modwsgi was related. >>> >>> FWIW, PyMongo master (aka 1.5.2+) works as expected everywhere. >> >> >> It would be odd that if it was a PyMongo issue that it only affected >> mod_wsgi 2.X. >> >> The only reason I could think for that is that how thread locals are >> managed in mod_wsgi changed between mod_wsgi 2.X and 3.X. >> >> Specifically, in mod_wsgi 2.X the thread local data was only preserved >> until the end of that request. >> >> In mod_wsgi 3.X the thread local data survives between requests using >> the same external thread from Apache's thread pool. >> >> Only problem that could come from this is if PyMongo has a C extension >> component which was coping properly with being destroyed all the time >> and would only work where the object persisted for the life of the >> process. >> >> That said, some pure Python WSGI servers don't have persistent thread >> pools and will destroy/create threads as required. Thus in that >> situation could also end up with thread local data being destroyed on >> a regular basis. > > I forgot to mention I'm running in Daemon mode. Does that affect your > explanation?
Initially I thought that it perhaps wasn't relevant. If though the problem also existed with mod_wsgi 3.X in embedded mode, albeit it not in daemon mode, then I would say possibly. This is because in mod_wsgi 3.X there was another subtle change in how daemon mode handled threads. In mod_wsgi 2.X, for both embedded mode and daemon mode, plus still in embedded mode of mod_wsgi 3.X, which thread is used out of the thread pool is random. Actually it depends on how the operating system thread libraries are implemented and how it determines which thread waiting on a mutex is woken up. In daemon mode of mod_wsgi 3.X the change was that the pool of threads is effectively managed as a stack. When a thread becomes free it is placed back on the top of the stack and when a new one is needed it is taken from the top of the stack. What this results in is that new requests will always use a recently used thread rather than one that hadn't been used in a while. This means that even though you might have default of 15 threads in thread pool, if you don't have many concurrent requests, it may only ever use the top couple of threads and all the other threads may never get used and may not even be initialised for use in Python code, thereby keeping per thread memory requirements down, unless the thread does actually need to be used. If there are never any concurrent requests, then you do still cycle through two threads and so only two thread local data objects involved. In embedded mode and older mod_wsgi, then it will eventually cycle through all threads in thread pool, so you may have many more active thread state objects. It just may be the case that the code didn't cause any problems when the number of activate thread state objects in relation to that class static was small, but did if it was large. As such, you may well have found that you did have problems if you managed to throw enough concurrent requests at mod_wsgi 3.X daemon mode to wake up more than just the top couple of threads on the thread pool stack. Anyway, although interesting technically, important thing is that the fix seems to address the problem. Graham >> If you have a reference to the specific bug fix that may have been >> made that addressed the issue, would be interested to see it. > > This is the fix: > > http://github.com/mongodb/mongo-python-driver/commit/dc3b2f73cf63c982afef5ca3cf27b43246781932#comments > > Pool.sockets was a class attribute instead of a non-thread-local > attribute on Pool instances. Therefore it was acting as a global and > sharing sockets between Pools (and therefore between Connections). > > Mike Dirolf's fix seems to have solved this issue entirely. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
