I recollect a couple of similar reports but not for a long time. I sort of remember them all involving use of older Apache prior to about 2.2.20. None with newer Apache versions.
Are you find even finding zombie processes for Apache hanging around? Is there a specific reason you need to use maximum-requests? Especially under load that is going to cause a huge churn of the processes, so if it is hitting some race condition as far as some problem in that very old Apache version, that wouldn’t help. Graham > On 4 Aug 2016, at 10:35 AM, Neil Sheth <[email protected]> wrote: > > Hello, > > We are subjecting our infrastructure to some load testing, and we're seeing > something curious. It appears that, after a few hours of running, wsgi > processes start failing to come back up. > > Here's what we're running: > Apache/2.2.15 (Unix) DAV/2 mod_wsgi/4.4.12 Python/2.7.6 configured > > Here's the wsgi config: > WSGISocketPrefix /var/run/wsgi > > NameVirtualHost *:8090 > > <VirtualHost *:8090> > ServerName loadtest.test.com > Timeout 1800 > > WSGIDaemonProcess testapp user=ep group=ep threads=1 processes=16 > python-path=/www/virtualenvs/testenv/lib/python2.7/site-packages > maximum-requests=1000 graceful-timeout=3600 socket-timeout=1800 > request-timeout=1800 > WSGIScriptAlias / /www/sites/app/app.wsgi > WSGIPassAuthorization On > > ErrorLog /www/logs/http/error_log > CustomLog /www/logs/http/access_log combined > > <Directory /www/sites/app> > WSGIProcessGroup perception > WSGIApplicationGroup %{GLOBAL} > WSGIChunkedRequest On > Order deny,allow > Allow from all > </Directory> > </VirtualHost> > > > In Apache: > Enter code here...<IfModule prefork.c> > StartServers 8 > MinSpareServers 5 > MaxSpareServers 20 > ServerLimit 400 > MaxClients 400 > MaxRequestsPerChild 1000 > </IfModule> > > GracefulShutDownTimeout 10 > > > In the apache error logs, we see things a constant stream of things like: > [Tue Aug 02 22:45:24 2016] [info] mod_wsgi (pid=23557): Process 'testapp' has > died, deregister and restart it. > [Tue Aug 02 22:45:24 2016] [info] mod_wsgi (pid=23557): Process 'testapp' > terminated normally, exit code 0 > [Tue Aug 02 22:45:24 2016] [info] mod_wsgi (pid=23557): Process 'testapp' has > been deregistered and will no longer be monitored. > [Tue Aug 02 22:45:24 2016] [info] mod_wsgi (pid=26067): Starting process > 'testapp' with uid=500, gid=500 and threads=1. > [Tue Aug 02 22:45:24 2016] [info] mod_wsgi (pid=26067): Initializing Python. > [Tue Aug 02 22:45:26 2016] [info] mod_wsgi (pid=20870): Destroying > interpreters. > [Tue Aug 02 22:45:26 2016] [info] mod_wsgi (pid=20870): Cleanup interpreter > ''. > [Tue Aug 02 22:45:26 2016] [info] mod_wsgi (pid=20870): Terminating Python. > [Tue Aug 02 22:45:26 2016] [info] mod_wsgi (pid=20870): Python has shutdown. > [Tue Aug 02 22:45:27 2016] [info] mod_wsgi (pid=23712): Process 'testapp' has > died, deregister and restart it. > [Tue Aug 02 22:45:27 2016] [info] mod_wsgi (pid=23712): Process 'testapp' > terminated normally, exit code 0 > [Tue Aug 02 22:45:27 2016] [info] mod_wsgi (pid=23712): Process 'testapp' has > been deregistered and will no longer be monitored. > [Tue Aug 02 22:45:27 2016] [info] mod_wsgi (pid=26121): Starting process > 'perception' with uid=500, gid=500 and threads=1. > [Tue Aug 02 22:45:27 2016] [info] mod_wsgi (pid=26121): Initializing Python. > [Tue Aug 02 22:45:33 2016] [info] mod_wsgi (pid=26224): Initializing Python. > [Tue Aug 02 22:45:33 2016] [info] mod_wsgi (pid=26224): Attach interpreter ''. > [Tue Aug 02 22:45:38 2016] [info] mod_wsgi (pid=20793): Destroying > interpreters. > [Tue Aug 02 22:45:38 2016] [info] mod_wsgi (pid=20793): Cleanup interpreter > ''. > [Tue Aug 02 22:45:38 2016] [info] mod_wsgi (pid=20793): Terminating Python. > [Tue Aug 02 22:45:38 2016] [info] mod_wsgi (pid=20793): Python has shutdown. > > And then, in our app error log, lots of things like: > [Wed Aug 03 10:00:47 2016] [error] [client 127.0.0.1] (11)Resource > temporarily unavailable: mod_wsgi (pid=10897): Unable to connect to WSGI > daemon process 'testapp' on '/var/run/wsgi.6940.0.1.sock' after multiple > attempts as listener backlog limit was exceeded. > [Wed Aug 03 10:00:48 2016] [error] [client 127.0.0.1] (11)Resource > temporarily unavailable: mod_wsgi (pid=10900): Unable to connect to WSGI > daemon process 'testapp' on '/var/run/wsgi.6940.0.1.sock' after multiple > attempts as listener backlog limit was exceeded. > [Wed Aug 03 10:00:48 2016] [error] [client 127.0.0.1] (11)Resource > temporarily unavailable: mod_wsgi (pid=10903): Unable to connect to WSGI > daemon process 'testapp' on '/var/run/wsgi.6940.0.1.sock' after multiple > attempts as listener backlog limit was exceeded. > [Wed Aug 03 10:00:49 2016] [error] [client 127.0.0.1] (11)Resource > temporarily unavailable: mod_wsgi (pid=9664): Unable to connect to WSGI > daemon process 'testapp' on '/var/run/wsgi.6940.0.1.sock' after multiple > attempts as listener backlog limit was exceeded. > > > One thing we noticed when we checked on it, while we expected 16 processes > running, only 1 actually was. Restarting apache brought them all back up, > but I haven't found any clues as to why this happens yet. Obviously, with > only one process running, throughput is severely decreased. > > Any insight would be appreciated! > > Thanks! > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To post to this group, send email to [email protected] > <mailto:[email protected]>. > Visit this group at https://groups.google.com/group/modwsgi > <https://groups.google.com/group/modwsgi>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
