On 15 June 2010 04:07, leenuxg33k <[email protected]> wrote: > I'm experiencing a strange one. I can't figure this out and I'm > hoping someone has either seen it or can tell me how to further debug > this problem. > > I have my tg1 app configured like so: > > WSGIDaemonProcess beaker user=apache group=apache display-name=beaker > maximum-requests=200 shutdown-timeout=15 threads=1 processes=7 > WSGISocketPrefix run/wsgi > # I've tried with the following on or off and it doesn't matter > #WSGIRestrictStdout On > #WSGIRestrictSignal Off > > WSGIPythonOptimize 2 > WSGIPassAuthorization On > > WSGIScriptAlias / /usr/share/bkr/beaker-server.wsgi > > <Directory /usr/share/bkr> > WSGIApplicationGroup beaker > WSGIProcessGroup beaker > Order deny,allow > Allow from all > </Directory> > > System environment: > [r...@beaker-stage ~]# rpm -q httpd > httpd-2.2.3-43.el5 > [r...@beaker-stage ~]# rpm -q mod_wsgi > mod_wsgi-3.2-1.el5 > [r...@beaker-stage ~]# more /etc/redhat-release > Red Hat Enterprise Linux Server release 5.5 (Tikanga) > [r...@beaker-stage ~]# uname -m > x86_64 > > > Everything works as expected for a while. but under heady load I > start to lose beaker processes. Eventually there will be none left. > Of course then I start getting 500 ISE's in apache log, but only after > long timeouts. > > I was able to run with LogLevel Info and have the following: > > [Sat Jun 12 08:06:56 2010] [info] mod_wsgi (pid=12312): Maximum > requests reached 'beaker'. > [Sat Jun 12 08:06:56 2010] [info] mod_wsgi (pid=12312): Shutdown > requested 'beaker'. > [Sat Jun 12 08:06:56 2010] [info] mod_wsgi (pid=12312): Stopping > process 'beaker'. > [Sat Jun 12 08:06:56 2010] [info] mod_wsgi (pid=12312): Destroying > interpreters. > [Sat Jun 12 08:06:56 2010] [info] mod_wsgi (pid=12312): Destroy > interpreter 'beaker'. > [Sat Jun 12 08:07:00 2010] [info] mod_wsgi (pid=12312): Cleanup > interpreter ''. > [Sat Jun 12 08:07:00 2010] [info] mod_wsgi (pid=12312): Terminating > Python. > [Sat Jun 12 08:07:00 2010] [info] mod_wsgi (pid=12312): Python has > shutdown. > [Sat Jun 12 08:07:00 2010] [info] mod_wsgi (pid=12312): Process > 'beaker' has died, restarting. > [Sat Jun 12 08:07:00 2010] [info] mod_wsgi (pid=12363): Starting > process 'beaker' with uid=48, gid=48 > and threads=1. > [Sat Jun 12 08:07:00 2010] [info] mod_wsgi (pid=12363): Initializing > Python. > [Sat Jun 12 08:07:00 2010] [info] mod_wsgi (pid=12363): Attach > interpreter ''. > [Sat Jun 12 08:07:00 2010] [info] mod_wsgi (pid=12363): Create > interpreter 'beaker'. > [Sat Jun 12 08:07:00 2010] [warn] mod_wsgi (pid=12363): Callback > registration for signal 15 ignored. > [Sat Jun 12 08:07:00 2010] [warn] File "/usr/share/bkr/beaker- > server.wsgi", line 10, in ? > [Sat Jun 12 08:07:00 2010] [warn] import cherrypy > [Sat Jun 12 08:07:00 2010] [warn] File "/usr/lib/python2.4/site- > packages/cherrypy/__init__.py", line > 20, in ? > [Sat Jun 12 08:07:00 2010] [warn] import _cpserver > [Sat Jun 12 08:07:00 2010] [warn] File "/usr/lib/python2.4/site- > packages/cherrypy/_cpserver.py", lin > e 10, in ? > [Sat Jun 12 08:07:00 2010] [warn] from cherrypy._cpengine import > Engine, STOPPED, STARTING, STARTE > D > [Sat Jun 12 08:07:00 2010] [warn] File "/usr/lib/python2.4/site- > packages/cherrypy/_cpengine.py", lin > e 25, in ? > [Sat Jun 12 08:07:00 2010] [warn] signal.signal(signal.SIGTERM, > SIGTERM) > > The traceback is normal I suspect because this time I was running with > the recommended settings to disable callbacks in cherypy. > > Here is the log when it stops spawning new processes.. > > [Sat Jun 12 08:07:03 2010] [info] mod_wsgi (pid=12363): Maximum > requests reached 'beaker'. > [Sat Jun 12 08:07:03 2010] [info] mod_wsgi (pid=12363): Shutdown > requested 'beaker'. > [Sat Jun 12 08:07:03 2010] [info] mod_wsgi (pid=12363): Stopping > process 'beaker'. > [Sat Jun 12 08:07:03 2010] [info] mod_wsgi (pid=12363): Destroying > interpreters. > [Sat Jun 12 08:07:03 2010] [info] mod_wsgi (pid=12363): Destroy > interpreter 'beaker'. > [Sat Jun 12 08:07:07 2010] [info] mod_wsgi (pid=12363): Cleanup > interpreter ''. > [Sat Jun 12 08:07:07 2010] [info] mod_wsgi (pid=12363): Terminating > Python. > [Sat Jun 12 08:07:07 2010] [info] mod_wsgi (pid=12363): Python has > shutdown. > [Sat Jun 12 08:12:04 2010] [info] mod_wsgi (pid=8920): Destroying > interpreters. > [Sat Jun 12 08:12:04 2010] [info] mod_wsgi (pid=8920): Cleanup > interpreter ''. > [Sat Jun 12 08:12:04 2010] [info] mod_wsgi (pid=8920): Terminating > Python. > [Sat Jun 12 08:12:04 2010] [info] mod_wsgi (pid=8920): Python has > shutdown. > [Sat Jun 12 08:12:05 2010] [info] mod_wsgi (pid=14184): Initializing > Python. > [Sat Jun 12 08:12:05 2010] [info] mod_wsgi (pid=14184): Attach > interpreter ''. > [Sat Jun 12 08:12:08 2010] [info] mod_wsgi (pid=14203): Initializing > Python. > [Sat Jun 12 08:12:08 2010] [info] mod_wsgi (pid=14203): Attach > interpreter ''. > [Sat Jun 12 08:50:26 2010] [error] [client 10.16.65.167] File does not > exist: /var/www/beaker/harness/ > noarch > [Sat Jun 12 08:50:27 2010] [info] mod_wsgi (pid=27862): Initializing > Python. > [Sat Jun 12 08:50:27 2010] [info] mod_wsgi (pid=27862): Attach > interpreter ''. > [Sat Jun 12 09:50:28 2010] [error] [client 10.16.65.167] File does not > exist: /var/www/beaker/harness/ > noarch > [Sat Jun 12 10:50:31 2010] [info] mod_wsgi (pid=5849): Initializing > Python. > [Sat Jun 12 10:50:31 2010] [info] mod_wsgi (pid=5849): Attach > interpreter ''. > [Sat Jun 12 10:50:32 2010] [info] mod_wsgi (pid=5855): Initializing > Python. > [Sat Jun 12 10:50:32 2010] [info] mod_wsgi (pid=5855): Attach > interpreter ''. > [Sat Jun 12 10:50:32 2010] [info] mod_wsgi (pid=5856): Initializing > Python. > [Sat Jun 12 10:50:32 2010] [info] mod_wsgi (pid=5856): Attach > interpreter ''. > [Sat Jun 12 11:12:36 2010] [info] mod_wsgi (pid=13739): Initializing > Python. > [Sat Jun 12 11:12:36 2010] [info] mod_wsgi (pid=13739): Attach > interpreter ''. > [Sat Jun 12 11:24:37 2010] [info] mod_wsgi (pid=18019): Initializing > Python. > [Sat Jun 12 11:24:37 2010] [info] mod_wsgi (pid=18019): Attach > interpreter ''. > [Sat Jun 12 11:24:38 2010] [info] mod_wsgi (pid=18026): Initializing > Python. > [Sat Jun 12 11:24:38 2010] [info] mod_wsgi (pid=18026): Attach > interpreter ''. > [Sat Jun 12 11:24:38 2010] [info] mod_wsgi (pid=18027): Initializing > Python. > [Sat Jun 12 11:24:38 2010] [info] mod_wsgi (pid=18027): Attach > interpreter ''. > [Sat Jun 12 11:36:38 2010] [info] mod_wsgi (pid=22305): Initializing > Python. > [Sat Jun 12 11:36:38 2010] [info] mod_wsgi (pid=22305): Attach > interpreter ''. > [Sat Jun 12 11:48:39 2010] [info] mod_wsgi (pid=26588): Initializing > Python. > [Sat Jun 12 11:48:39 2010] [info] mod_wsgi (pid=26588): Attach > interpreter ''. > [Sat Jun 12 11:48:40 2010] [info] mod_wsgi (pid=26595): Initializing > Python. > [Sat Jun 12 11:48:40 2010] [info] mod_wsgi (pid=26596): Initializing > Python. > [Sat Jun 12 11:48:40 2010] [info] mod_wsgi (pid=26596): Attach > interpreter ''. > [Sat Jun 12 11:48:40 2010] [info] mod_wsgi (pid=26595): Attach > interpreter ''. > > etc... > > Any ideas? No OomKills or any segfaults at the time of these > problems.
Run: httpd -V if you are using prefork MPM, then switch to worker MPM to see if it makes a difference. If you are using mod_php and so cant do that, then you may be stuck. Anyway, one other person has noted problems like what you are seeing when Apache prefork MPM is being used. The problem appeared to go away when worker MPM was used. There may therefore be some issue in Apache with process management as mod_wsgi relies on process Apache management code and whether that works properly or not may be affected by which MPM is used. It isn't known yet whether this might only be related to specific Apache/APR versions. The other thing that can be done to at least lesson possibility of it occurring is not use maximum-requests option to WSGIDaemonProcess. If your TG application is not releasing objects properly and growing in memory size over time, you may not be able to do that. Graham -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
