The problem persists. I have removed our calls to lxml; they were not critical. We'll see what effect that has going forward.
-Chase On Apr 16, 12:08 am, Graham Dumpleton <[email protected]> wrote: > On 16 April 2011 01:04, Chase <[email protected]> wrote: > > > > > > > > > > > Wow, lots of good info. Thanks guys! I have made the > > "WSGIApplicationGroup %{GLOBAL}" change for now; we'll see if that > > clears it up over the next week or so. > > > As for running in prefork, I have not made that change yet. But here > > is the documentation that lead me to believe this was preferred: > > >http://code.google.com/p/modwsgi/wiki/IntegrationWithDjango > > > "Now, traditional wisdom in respect of Django has been that it should > > perferably only be used on single threaded servers. This would mean > > for Apache using the single threaded 'prefork' MPM on UNIX systems and > > avoiding the multithreaded 'worker' MPM." > > > Also, the older modpython docs also advised this: > > >http://docs.djangoproject.com/en/dev/howto/deployment/modpython/?from... > > > "Django requires Apache 2.x and mod_python 3.x, and you should use > > Apache’s prefork MPM, as opposed to the worker MPM." > > > Can you link to a discussion of the subtle problems reported with > > prefork? Thanks again, > > That section was more relevant when Django 1.0 had only just come out, > which was the first version of Django for which the core was > supposedly thread safe. > > Anyway, the MPM you use isn't particularly relevant as you are using > daemon mode and not embedded mode. Which MPM you use is only critical > if you are using embedded mode. > > In daemon mode you have the arbitrary ability to control > processes/threads based on whether your application is thread safe. > > For related reading see: > > http://code.google.com/p/modwsgi/wiki/ProcessesAndThreading > http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usa... > > BTW, the IntegrationWithDjango page in the wiki is likely to be > completely removed at some point in the near future and I will stop > providing details for specific frameworks to cover where frameworks > don't themselves provide enough information. I have already removed > the pages for most of the other frameworks already. End result is that > the frameworks themselves will need to provide decent documentation > themselves to cover any idiosyncrasies that exist in setting up their > framework to work with mod_wsgi which are due to issues or design > decisions related to their framework and which are nothing to do with > mod_wsgi. I have had enough of trying to document these framework > specific subtleties and framework authors tend to express a belief > that their own documentation is already more than adequate even though > from what I have seen people still get tripped up when they follow > only the documentation provided by the framework. So, I will be > devoting my time elsewhere now and not worrying about documenting > stuff related to the frameworks or actively assisting users of > frameworks on forums related to those frameworks or on general forums > such as StackOverflow. Instead, if it is a framework specific issue, > you will need to seek help from the developers or the community for > that framework. > > Graham > > > > > > > > > -Chase > > > On Apr 14, 6:30 pm, Graham Dumpleton <[email protected]> > > wrote: > >> On 15 April 2011 05:18, Chase <[email protected]> wrote: > > >> > I have a custom Django app that's becoming unresponsive > >> > intermittently. About once every couple of days between three servers, > >> > serving about 10,000 requests a day. When it happens, it never > >> > recovers. I can leave it there for hours, and it will not server any > >> > more requests. > > >> > In the apache logs, I see see the following: > > >> > Apr 13 11:45:07 www3 apache2[27590]: **successful view render here** > >> > ... > >> > Apr 13 11:47:11 www3 apache2[24032]: [error] server is within > >> > MinSpareThreads of MaxClients, consider raising the MaxClients setting > >> > Apr 13 11:47:43 www3 apache2[24032]: [error] server reached MaxClients > >> > setting, consider raising the MaxClients setting > >> > ... > >> > Apr 13 11:50:34 www3 apache2[27617]: [error] [client 10.177.0.204] > >> > Script timed out before returning headers: django.wsgi > >> > (repeated 100 times, exactly) > > >> > I am running: > > >> > apache version 2.2, using the worker MPM > >> > wsgi version 2.8 > >> > SELinux NOT installed > >> > lxml package being used, infrequently > >> > Ubuntu 10.04 > > >> > apache config: > > >> > WSGIDaemonProcess site-1 user=django group=django threads=50 > >> > WSGIProcessGroup site-1 > >> > WSGIScriptAlias / /somepath/django.wsgi /somepath/django.wsgi > > >> > wsgi config: > > >> > import os, sys > >> > sys.path.append('/home/django') > >> > os.environ['DJANGO_SETTINGS_MODULE'] = 'myapp.settings' > >> > import django.core.handlers.wsgi > >> > application = django.core.handlers.wsgi.WSGIHandler() > > >> > When this happens, I can kill the wsgi process and the server will > >> > recover. > > >> >>ps aux|grep django # process is running as user "django" > >> > django 27590 5.3 17.4 908024 178760 ? Sl Apr12 76:09 /usr/ > >> > sbin/apache2 -k start > >> >>kill -9 27590 > > >> > This leads me to believe that the problem is a known issue: > > >> > "(deadlock-timeout) Defines the maximum number of seconds allowed to > >> > pass before the daemon process is shutdown and restarted after a > >> > potential deadlock on the Python GIL has been detected. The default is > >> > 300 seconds. This option exists to combat the problem of a daemon > >> > process freezing as the result of a rouge Python C extension module > >> > which doesn't properly release the Python GIL when entering into a > >> > blocking or long running operation." > > >> > However, I'm not sure why this condition is not clearing > >> > automatically. I do see that the script timeout occurs exactly 5 > >> > minutes after the last successful page render, so the deadlock-timeout > >> > is getting triggered. But it does not actually kill the process. > > >> They likely aren't being killed because there isn't actually a > >> deadlock of a single thread which hasn't release the GIL. > > >> In other words, what the dead lock timeout will not protect against is > >> threads calling into C code, releasing the GIL and then deadlocking in > >> C code. > > >> In your case, the problem is going to be the lxml module. This module > >> is known not to work in Python sub interpreters properly. > >> Specifically, the lxml can release the GIL and then attempt to do a > >> callback into Python code. To do this, it uses the simplified GIL > >> state API in Python to reacquire the GIL, but that API is only > >> supposed to be used if running in the main Python interpreter and not > >> a sub interpreter. When used in a sub interpreter, the code will > >> deadlock on trying to reacquire the Python GIL. > > >> That lxml is a problem is documented in: > > >> http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Multiple_Pyth... > > >> The solution, since you are only delegating one application to that > >> mod_wsgi daemon process group, is to add: > > >> WSGIApplicationGroup %{GLOBAL} > > >> This will force the application to run in the main Python interpreter > >> and avoid the shortcomings of lxml module. > > >> As how you might protect against this sort of deadlock in C code when > >> GIL isn't locked, the only way is to use 'inactivity-timeout'. This > >> will cause a restart when there has been no new requests and/or no > >> reading of request content or generation of response content for that > >> timeout period. So, this could be used as a fail safe, but if your > >> application is used in frequently, it will also have the affect of > >> causing your idle process to be restarted after the timeout period as > >> well. > > >> BTW, in worst cases, for detecting what process is doing, one can use > >> either: > > >> http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Extracting_... > >> http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Debugging_C... > > >> > I'm thinking of switching to MPM/prefork, but I'm not sure if that > >> > should have any effect, given that I'm in daemon mode already. > > >> Prefork for some people has been causing subtle problems and I would > >> avoid it if you can. > > >> Graham > > > -- > > You received this message because you are subscribed to the Google Groups > > "modwsgi" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group > > athttp://groups.google.com/group/modwsgi?hl=en. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
