Before I can help you, you need to start answering my questions. If you pepper me with email after email on different things and don't answer the questions I ask you, then we will get no where. When I ask questions it is because that information is likely key to me being able to understand the problem and help you.
The main question I put to you was about how you are determining which processes are growing in memory. It needs to be properly determined whether the processes which were growing in memory were Apache child worker processes or mod_wsgi daemon processes. The reason for this is that if they are in fact mod_wsgi daemon processes, then you are possibly in part causing the problem yourself by setting MaxConnectionsPerChild. That setting has got nothing to do with the mod_wsgi daemon processes and by setting it, you are creating the conditions which cause the sorts of errors you are seeing. So, which processes are growing in memory? Graham On 31/10/2014, at 12:40 AM, Kelvin Wong <[email protected]> wrote: > I found that on the main Apache log, when it is occurring, Apache is doing > graceful restart. > How can I avoid Apache graceful restart? > > Thanks. > > > On Tuesday, October 28, 2014 1:30:06 PM UTC+8, Graham Dumpleton wrote: > Would suggest upgrading to mod_wsgi 4.3.0 if you can as the error messages > when there are communication problems between Apache child worker process and > mod_wsgi daemon process have been improved. > > More comments below. > > On 28 October 2014 15:43, Kelvin Wong <[email protected]> wrote: > Hi Graham and everyone else > > I'm running multiple site on Django 1.6.7, Apache/2.4.7 (Ubuntu 14.04), > OpenSSL/1.0.1f, mod_wsgi/4.2.5, Python/2.7.6, Server MPM: worker. > I found that the server start returning 504 and then 503, and the following > error shown up. > I researched some issues related with it, even added "WSGISocketPrefix > /var/run/apache2/wsgi", but the issue still occured. > I have no idea why it happened. Can anyone give some directions on this issue? > > Thanks! > > apache error log > [Sun Oct 26 07:34:34.732934 2014] [wsgi:error] [pid 29268:tid > 140053011478272] [client xx.xxx.xxx.xxx:xxxxx] Timeout when reading response > headers from daemon process 'site-1': /home/ubuntu/site-1/apache/wsgi.py > [Sun Oct 26 07:34:37.198806 2014] [wsgi:error] [pid 27816:tid > 140052910765824] (11)Resource temporarily unavailable: [client > xx.xx.xx.xx:xxxxx] mod_wsgi (pid=27816): Unable to connect to WSGI daemon > process 'site-1' on '/var/run/apache2/wsgi.17227.2.3.sock'. > > This one can occur when the mod_wsgi daemon process crashes. There should be > a segmentation fault error message or similar in the main Apache error log > (not VirtualHost specific log). > > It can also occur if there are incomplete requests still running when a > mod_wsgi daemon process is shutdown on being restarted due to the WSGI script > file being touched or if Apache was restarted. In the latter case, the > mod_wsgi daemon process would have had to have been killed off by Apache > before the Apache child worker process which was proxying it to had. This can > especially be the case if an Apache graceful restart was being done. > > occasionally > [Tue Oct 28 02:20:40.722140 2014] [wsgi:error] [pid 24158:tid > 140182690981632] (2)No such file or directory: [client 24.171.250.159:60769] > mod_wsgi (pid=24158): Unable to connect to WSGI daemon process > 'snaptee-production-api-ssl' on '/var/run/apache2/wsgi.30188.7.3.sock'. > > This can also be due to Apache graceful restart being done and there were > keep alive connections being handled from a HTTP client. In an Apache > graceful restart, because of Apache handles the mod_wsgi daemon processes, > they don't have a graceful shutdown in the same way as Apache child worker > processes. > > So what happens is the following: > > 1. Apache graceful restart is triggered. > 2. Apache parent process sends SIGUSR1 to Apache child worker process to > signal graceful shutdown. > 3. Apache parent process sends SIGINT to mod_wsgi daemon processes to signal > shutdown. > 4. The mod_wsgi daemon processes complete their requests and restart. In the > next incarnation of the mod_wsgi daemon processes after an Apache restart > they expect a different path for the proxy socket, with the number at the end > increasing based on Apache generation number. > 5. The Apache child worker process because it was in a graceful restart mode, > operates on the understanding that it can keep handling any requests on a > keep alive socket connection from a HTTP client until there are no more. It > therefore takes next request on same connection and tries to connect to > mod_wsgi daemon process, but using the proxy socket name as was used before, > but that name has changed for the next Apache configuration generation and no > longer exists, thus it fails. > > The name of the proxy socket changes across Apache restarts because otherwise > you could have Apache child worker processes under an old configuration > sending requests to a mod_wsgi daemon process using the new configuration, > which could cause problems including security issues. There are therefore > specific protections in place to ensure that only Apache child worker > processes and mod_wsgi daemon mode processes created against the same Apache > configuration generation can talk to each other. > > wsgi config for that site > WSGIDaemonProcess site-1 display-name=site-1 user=www-data threads=25 > python-path=/home/ubuntu/site-1/django:/home/ubuntu/.virtualenvs/site-1/lib/python2.7/site-packages > WSGIProcessGroup site-1 > WSGIApplicationGroup %{GLOBAL} > WSGIScriptAlias / /home/ubuntu/site-1/apache/wsgi.py > > worker.conf > <IfModule mpm_worker_module> > StartServers 2 > MinSpareThreads 25 > MaxSpareThreads 75 > ThreadLimit 64 > ThreadsPerChild 25 > MaxRequestWorkers 150 > MaxConnectionsPerChild 1000 > </IfModule> > > So my best guess is that you are doing Apache graceful restarts when these > are occurring. > > Are you using Apache graceful restarts as suspected? > > Graham > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/modwsgi. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
