Don't know of any problems with ElementTree. >From memory, the problem with lxml is because it uses SWIG to generate Python wrappers for C internals and it is SWIG that uses simplified GIL state API when doing callbacks.
Thus, this problem generally affects anything that uses SWIG and which is doing callbacks from C code into Python. Even if my memory is bad and lxml doesn't use SWIG, the issue with SWIG still stands. Graham On 15 April 2011 09:04, Carl Nobile <[email protected]> wrote: > I didn't know that lxml was an issue. I will keep that in mind. Does the C > version of ElementTree have the same issues? > Though lxml is a better package because it implements more of the XML spec > like true xpath and XSLT. > > ~Carl > > > On Thu, Apr 14, 2011 at 6:38 PM, Graham Dumpleton > <[email protected]> wrote: >> >> On 15 April 2011 07:09, Graham Dumpleton <[email protected]> >> wrote: >> > On 15 April 2011 06:35, Carl Nobile <[email protected]> wrote: >> >> It looks like you are running a single process with 50 threads, I think >> >> you >> >> should use more processes with less threads something like this: >> >> >> >> WSGIDaemonProcess site-1 user=django group=django processes=5 >> >> threads=10 >> >> maximum-requests=1000 >> >> WSGIProcessGroup site-1 >> >> WSGIScriptAlias / /somepath/django.wsgi /somepath/django.wsgi >> >> >> >> The 'maximum-requests=1000' will kill each thread after a 1000 requests >> >> and >> >> create a new one, this helps to keep memory leaks to a minimum. >> > >> > Sorry to correct you Carl, but that isn't quite how it works. >> >> The small correction is that once that number of threads is reached >> for the whole process, irrespective of how many threads are running in >> the process, then the process as a whole is killed off and restarted. >> It isn't done at individual thread level within ongoing process. >> >> The maximum-requests option should be avoided in production processes >> if at all possible because the quicker requests come through the more >> frequently the process will restart, which is likely the last thing >> you want to happen when under load. >> >> As to number of processes/threads, as Carl pointed out, OP should >> avoid having high numbers of threads in a single process and instead >> create multiple processes with a small number of threads. >> >> For most people, the default of 15 threads per process is likely >> overkill with that many concurrent requests never actually occurring, >> so increasing it with no good reason is not a good idea. If you have >> the memory available, possibly better off going to 3 processes each >> with 5 threads only. >> >> Graham >> >> > I'll respond in more detail later to original question. Still 7am here >> > and just got off phone from a work meeting. So need to wake up a bit >> > more first. :-) >> > >> > Graham >> > >> >> ~Carl >> >> >> >> On Thu, Apr 14, 2011 at 3:18 PM, Chase <[email protected]> wrote: >> >>> >> >>> I have a custom Django app that's becoming unresponsive >> >>> intermittently. About once every couple of days between three servers, >> >>> serving about 10,000 requests a day. When it happens, it never >> >>> recovers. I can leave it there for hours, and it will not server any >> >>> more requests. >> >>> >> >>> >> >>> In the apache logs, I see see the following: >> >>> >> >>> Apr 13 11:45:07 www3 apache2[27590]: **successful view render here** >> >>> ... >> >>> Apr 13 11:47:11 www3 apache2[24032]: [error] server is within >> >>> MinSpareThreads of MaxClients, consider raising the MaxClients setting >> >>> Apr 13 11:47:43 www3 apache2[24032]: [error] server reached MaxClients >> >>> setting, consider raising the MaxClients setting >> >>> ... >> >>> Apr 13 11:50:34 www3 apache2[27617]: [error] [client 10.177.0.204] >> >>> Script timed out before returning headers: django.wsgi >> >>> (repeated 100 times, exactly) >> >>> >> >>> >> >>> I am running: >> >>> >> >>> apache version 2.2, using the worker MPM >> >>> wsgi version 2.8 >> >>> SELinux NOT installed >> >>> lxml package being used, infrequently >> >>> Ubuntu 10.04 >> >>> >> >>> >> >>> apache config: >> >>> >> >>> WSGIDaemonProcess site-1 user=django group=django threads=50 >> >>> WSGIProcessGroup site-1 >> >>> WSGIScriptAlias / /somepath/django.wsgi /somepath/django.wsgi >> >>> >> >>> >> >>> wsgi config: >> >>> >> >>> import os, sys >> >>> sys.path.append('/home/django') >> >>> os.environ['DJANGO_SETTINGS_MODULE'] = 'myapp.settings' >> >>> import django.core.handlers.wsgi >> >>> application = django.core.handlers.wsgi.WSGIHandler() >> >>> >> >>> >> >>> When this happens, I can kill the wsgi process and the server will >> >>> recover. >> >>> >> >>> >ps aux|grep django # process is running as user "django" >> >>> django 27590 5.3 17.4 908024 178760 ? Sl Apr12 76:09 /usr/ >> >>> sbin/apache2 -k start >> >>> >kill -9 27590 >> >>> >> >>> >> >>> This leads me to believe that the problem is a known issue: >> >>> >> >>> "(deadlock-timeout) Defines the maximum number of seconds allowed to >> >>> pass before the daemon process is shutdown and restarted after a >> >>> potential deadlock on the Python GIL has been detected. The default is >> >>> 300 seconds. This option exists to combat the problem of a daemon >> >>> process freezing as the result of a rouge Python C extension module >> >>> which doesn't properly release the Python GIL when entering into a >> >>> blocking or long running operation." >> >>> >> >>> >> >>> However, I'm not sure why this condition is not clearing >> >>> automatically. I do see that the script timeout occurs exactly 5 >> >>> minutes after the last successful page render, so the deadlock-timeout >> >>> is getting triggered. But it does not actually kill the process. >> >>> >> >>> I'm thinking of switching to MPM/prefork, but I'm not sure if that >> >>> should have any effect, given that I'm in daemon mode already. >> >>> >> >>> -- >> >>> You received this message because you are subscribed to the Google >> >>> Groups >> >>> "modwsgi" group. >> >>> To post to this group, send email to [email protected]. >> >>> To unsubscribe from this group, send email to >> >>> [email protected]. >> >>> For more options, visit this group at >> >>> http://groups.google.com/group/modwsgi?hl=en. >> >>> >> >> >> >> >> >> >> >> -- >> >> >> >> ------------------------------------------------------------------------------- >> >> Carl J. Nobile (Software Engineer) >> >> [email protected] >> >> >> >> ------------------------------------------------------------------------------- >> >> >> >> -- >> >> You received this message because you are subscribed to the Google >> >> Groups >> >> "modwsgi" group. >> >> To post to this group, send email to [email protected]. >> >> To unsubscribe from this group, send email to >> >> [email protected]. >> >> For more options, visit this group at >> >> http://groups.google.com/group/modwsgi?hl=en. >> >> >> > >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/modwsgi?hl=en. >> > > > > -- > ------------------------------------------------------------------------------- > Carl J. Nobile (Software Engineer) > [email protected] > ------------------------------------------------------------------------------- > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/modwsgi?hl=en. > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
