Hi Graham, Thanks a lot for your detailed explanations.
I used to reload the apache processes instead of restart them. So is there any relation to the "MaxConnectionsPerChild" setting that when the process met the limit, it restart the child process? If so, any alternative to this setting? I used this setting to bound the memory usage of apache. Upgrading to mod_wsgi 4.3.0 will solve this problem? mod_wsgi 4.3.0 improved handling on segmentation fault error? On Tuesday, October 28, 2014 1:30:06 PM UTC+8, Graham Dumpleton wrote: > > Would suggest upgrading to mod_wsgi 4.3.0 if you can as the error messages > when there are communication problems between Apache child worker process > and mod_wsgi daemon process have been improved. > > More comments below. > > On 28 October 2014 15:43, Kelvin Wong <[email protected] <javascript:>> > wrote: > >> Hi Graham and everyone else >> >> I'm running multiple site on Django 1.6.7, Apache/2.4.7 (Ubuntu 14.04), >> OpenSSL/1.0.1f, mod_wsgi/4.2.5, Python/2.7.6, Server MPM: worker. >> I found that the server start returning 504 and then 503, and the >> following error shown up. >> I researched some issues related with it, even added "WSGISocketPrefix >> /var/run/apache2/wsgi", but the issue still occured. >> I have no idea why it happened. Can anyone give some directions on this >> issue? >> >> Thanks! >> >> apache error log >> [Sun Oct 26 07:34:34.732934 2014] [wsgi:error] [pid 29268:tid >> 140053011478272] [client xx.xxx.xxx.xxx:xxxxx] Timeout when reading >> response headers from daemon process 'site-1': /home/ubuntu/site-1/apache >> /wsgi.py >> [Sun Oct 26 07:34:37.198806 2014] [wsgi:error] [pid 27816:tid >> 140052910765824] (11)Resource temporarily unavailable: [client xx.xx.xx. >> xx:xxxxx] mod_wsgi (pid=27816): Unable to connect to WSGI daemon process >> 'site-1' on '/var/run/apache2/wsgi.17227.2.3.sock'. >> > > This one can occur when the mod_wsgi daemon process crashes. There should > be a segmentation fault error message or similar in the main Apache error > log (not VirtualHost specific log). > > It can also occur if there are incomplete requests still running when a > mod_wsgi daemon process is shutdown on being restarted due to the WSGI > script file being touched or if Apache was restarted. In the latter case, > the mod_wsgi daemon process would have had to have been killed off by > Apache before the Apache child worker process which was proxying it to had. > This can especially be the case if an Apache graceful restart was being > done. > > >> occasionally >> [Tue Oct 28 02:20:40.722140 2014] [wsgi:error] [pid 24158:tid >> 140182690981632] (2)No such file or directory: [client 24.171.250.159: >> 60769] mod_wsgi (pid=24158): Unable to connect to WSGI daemon process >> 'snaptee-production-api-ssl' on '/var/run/apache2/wsgi.30188.7.3.sock'. >> > > This can also be due to Apache graceful restart being done and there were > keep alive connections being handled from a HTTP client. In an Apache > graceful restart, because of Apache handles the mod_wsgi daemon processes, > they don't have a graceful shutdown in the same way as Apache child worker > processes. > > So what happens is the following: > > 1. Apache graceful restart is triggered. > 2. Apache parent process sends SIGUSR1 to Apache child worker process to > signal graceful shutdown. > 3. Apache parent process sends SIGINT to mod_wsgi daemon processes to > signal shutdown. > 4. The mod_wsgi daemon processes complete their requests and restart. In > the next incarnation of the mod_wsgi daemon processes after an Apache > restart they expect a different path for the proxy socket, with the number > at the end increasing based on Apache generation number. > 5. The Apache child worker process because it was in a graceful restart > mode, operates on the understanding that it can keep handling any requests > on a keep alive socket connection from a HTTP client until there are no > more. It therefore takes next request on same connection and tries to > connect to mod_wsgi daemon process, but using the proxy socket name as was > used before, but that name has changed for the next Apache configuration > generation and no longer exists, thus it fails. > > The name of the proxy socket changes across Apache restarts because > otherwise you could have Apache child worker processes under an old > configuration sending requests to a mod_wsgi daemon process using the new > configuration, which could cause problems including security issues. There > are therefore specific protections in place to ensure that only Apache > child worker processes and mod_wsgi daemon mode processes created against > the same Apache configuration generation can talk to each other. > > >> wsgi config for that site >> WSGIDaemonProcess site-1 display-name=site-1 user=www-data threads=25 >> python-path=/home/ubuntu/site-1/django:/home/ubuntu/.virtualenvs/site-1/ >> lib/python2.7/site-packages >> WSGIProcessGroup site-1 >> WSGIApplicationGroup %{GLOBAL} >> WSGIScriptAlias / /home/ubuntu/site-1/apache/wsgi.py >> >> worker.conf >> <IfModule mpm_worker_module> >> StartServers 2 >> MinSpareThreads 25 >> MaxSpareThreads 75 >> ThreadLimit 64 >> ThreadsPerChild 25 >> MaxRequestWorkers 150 >> MaxConnectionsPerChild 1000 >> </IfModule> >> > > So my best guess is that you are doing Apache graceful restarts when these > are occurring. > > Are you using Apache graceful restarts as suspected? > > Graham > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
