Hello,
We are subjecting our infrastructure to some load testing, and we're seeing
something curious. It appears that, after a few hours of running, wsgi
processes start failing to come back up.
Here's what we're running:
Apache/2.2.15 (Unix) DAV/2 mod_wsgi/4.4.12 Python/2.7.6 configured
Here's the wsgi config:
WSGISocketPrefix /var/run/wsgi
NameVirtualHost *:8090
<VirtualHost *:8090>
ServerName loadtest.test.com
Timeout 1800
WSGIDaemonProcess testapp user=ep group=ep threads=1 processes=16
python-path=/www/virtualenvs/testenv/lib/python2.7/site-packages
maximum-requests=1000 graceful-timeout=3600 socket-timeout=1800
request-timeout=1800
WSGIScriptAlias / /www/sites/app/app.wsgi
WSGIPassAuthorization On
ErrorLog /www/logs/http/error_log
CustomLog /www/logs/http/access_log combined
<Directory /www/sites/app>
WSGIProcessGroup perception
WSGIApplicationGroup %{GLOBAL}
WSGIChunkedRequest On
Order deny,allow
Allow from all
</Directory>
</VirtualHost>
In Apache:
Enter code here...<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 400
MaxClients 400
MaxRequestsPerChild 1000
</IfModule>
GracefulShutDownTimeout 10
In the apache error logs, we see things a constant stream of things like:
[Tue Aug 02 22:45:24 2016] [info] mod_wsgi (pid=23557): Process 'testapp'
has died, deregister and restart it.
[Tue Aug 02 22:45:24 2016] [info] mod_wsgi (pid=23557): Process 'testapp'
terminated normally, exit code 0
[Tue Aug 02 22:45:24 2016] [info] mod_wsgi (pid=23557): Process 'testapp'
has been deregistered and will no longer be monitored.
[Tue Aug 02 22:45:24 2016] [info] mod_wsgi (pid=26067): Starting process
'testapp' with uid=500, gid=500 and threads=1.
[Tue Aug 02 22:45:24 2016] [info] mod_wsgi (pid=26067): Initializing Python.
[Tue Aug 02 22:45:26 2016] [info] mod_wsgi (pid=20870): Destroying
interpreters.
[Tue Aug 02 22:45:26 2016] [info] mod_wsgi (pid=20870): Cleanup interpreter
''.
[Tue Aug 02 22:45:26 2016] [info] mod_wsgi (pid=20870): Terminating Python.
[Tue Aug 02 22:45:26 2016] [info] mod_wsgi (pid=20870): Python has shutdown.
[Tue Aug 02 22:45:27 2016] [info] mod_wsgi (pid=23712): Process 'testapp'
has died, deregister and restart it.
[Tue Aug 02 22:45:27 2016] [info] mod_wsgi (pid=23712): Process 'testapp'
terminated normally, exit code 0
[Tue Aug 02 22:45:27 2016] [info] mod_wsgi (pid=23712): Process 'testapp'
has been deregistered and will no longer be monitored.
[Tue Aug 02 22:45:27 2016] [info] mod_wsgi (pid=26121): Starting process
'perception' with uid=500, gid=500 and threads=1.
[Tue Aug 02 22:45:27 2016] [info] mod_wsgi (pid=26121): Initializing Python.
[Tue Aug 02 22:45:33 2016] [info] mod_wsgi (pid=26224): Initializing Python.
[Tue Aug 02 22:45:33 2016] [info] mod_wsgi (pid=26224): Attach interpreter
''.
[Tue Aug 02 22:45:38 2016] [info] mod_wsgi (pid=20793): Destroying
interpreters.
[Tue Aug 02 22:45:38 2016] [info] mod_wsgi (pid=20793): Cleanup interpreter
''.
[Tue Aug 02 22:45:38 2016] [info] mod_wsgi (pid=20793): Terminating Python.
[Tue Aug 02 22:45:38 2016] [info] mod_wsgi (pid=20793): Python has shutdown.
And then, in our app error log, lots of things like:
[Wed Aug 03 10:00:47 2016] [error] [client 127.0.0.1] (11)Resource
temporarily unavailable: mod_wsgi (pid=10897): Unable to connect to WSGI
daemon process 'testapp' on '/var/run/wsgi.6940.0.1.sock' after multiple
attempts as listener backlog limit was exceeded.
[Wed Aug 03 10:00:48 2016] [error] [client 127.0.0.1] (11)Resource
temporarily unavailable: mod_wsgi (pid=10900): Unable to connect to WSGI
daemon process 'testapp' on '/var/run/wsgi.6940.0.1.sock' after multiple
attempts as listener backlog limit was exceeded.
[Wed Aug 03 10:00:48 2016] [error] [client 127.0.0.1] (11)Resource
temporarily unavailable: mod_wsgi (pid=10903): Unable to connect to WSGI
daemon process 'testapp' on '/var/run/wsgi.6940.0.1.sock' after multiple
attempts as listener backlog limit was exceeded.
[Wed Aug 03 10:00:49 2016] [error] [client 127.0.0.1] (11)Resource
temporarily unavailable: mod_wsgi (pid=9664): Unable to connect to WSGI
daemon process 'testapp' on '/var/run/wsgi.6940.0.1.sock' after multiple
attempts as listener backlog limit was exceeded.
One thing we noticed when we checked on it, while we expected 16 processes
running, only 1 actually was. Restarting apache brought them all back up,
but I haven't found any clues as to why this happens yet. Obviously, with
only one process running, throughput is severely decreased.
Any insight would be appreciated!
Thanks!
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.
