On 8 April 2010 22:35, Sharjeel Ahmed Qureshi <[email protected]> wrote: > There is no particular reason for using Prefork except it is the default > with apache and I've heard it is safer than Worker. We are not using mod_php > at all.
Prefork is only required if using something like mod_php as it is single threaded. If all you are using the Apache for is to use mod_wsgi and not doing anything else, quite okay and much better to use worker MPM. There are no reliability issues and that is multithreaded will not matter to Python code as not running Python code in the Apache server child processes which the MPM controls. > Earlier I was getting a warning that wsgi is compiled for Python 2.5.1 but > Python2.5.2 is being used; I fixed it by re-compiling mod_wsgi to an > upgraded version. Shouldn't matter so long as Python using shared library. > My application uses a lot of packages. I'll have to make a list to see which > of them use C extension. > > I'm not sure about that 5 minute window; when my application goes down it > says down for hours until manually apache's spawned processes are killed and > apache is restarted. If 5 minute is the default timeout, then atleast > application should get back in some amount of time (be that hours). That is only the timeout for a Python GIL deadlock. The 'inactivity-timeout' option I mentioned is another more general timeout value you can specify. What it does is that if no requests at all for the specified period, will restart daemon process just to take memory back to base level. It also though monitors active requests and if all active requests don't consume request content, or generate response content for the specified period, will assume things are hung and trigger a process restart. This latter situation should pick up your situation unless the whole process is borked in some way to extent that even the pure C background thread which detects these conditions cant run. It would take something pretty bad for that too happen such as resource exhaustion. If that doesn't help, then have seen one other case where for some reason it looked like Linux itself was stuffing up. For the socket connection to daemon process, Apache server process was seeing a successful connect message, but listener in daemon process was never seeing a socket connection request. I made a change in mod_wsgi 2.7 related to timeouts on that connect though to at least recover from the problem. I would expect slightly different error messages though if that was occuring. It could though result in the daemon process appearing not to hang as seems that when process gets in that stuffed state, that it never recovers and all requests simply keep failing after the connect timeout. Are you seeing any error messages related to a socket timeout? Maybe if you can bundle up some complete examples of Apache error log files and email them to my personal email I can look at it. That way may pick up other error messages which you might not think are related but are. Given that have heard of that odd behaviour once before, would be good to track it down a bit more. Last time it was quite hard to even work that much out as like know was relying on person with problem to give me what they thought was important from logs. If I can get the whole logs will help immensely as can look for all sorts of messages and also look at the timing between different messages in error logs to perhaps work out what is going on. > There is one more thing; for certain task we spawn a python thread which > does some task in background (e.g. sending mail through Yahoo/GMail) while > it immediately gives back response to the user. Could that be a problem? Hard to tell. Graham > On Thu, Apr 8, 2010 at 5:22 PM, Graham Dumpleton > <[email protected]> wrote: >> >> On 8 April 2010 22:13, Sharjeel Ahmed Qureshi >> <[email protected]> wrote: >> > Graham, >> > >> > Thanks for the info. I have checked both logs, core as well as the >> > vhost's >> > logs. Yes, I've been lately getting this as well but they weren't >> > appearing >> > few days back, although problem has been appearing for the past many >> > days: >> > >> > [info] mod_wsgi (pid=xxxxx): Daemon process deadlock timer expired, >> > stopping >> > process 'seenreport'. >> >> If you are getting that then it would normally mean that some Python >> extension module (ie., C code), is blocking indefinitely but not >> releasing the Python GIL while doing so. This causes all request >> threads to lock up. By default this timeout occurs after 5 minutes and >> is basically a fail safe to cope with buggy extension modules. >> >> What third party Python packages are you using which have C extension >> modules as part of them? >> >> Another possible cause is mismatches in shared library versions used >> by Apache, Apache modules and Python modules. >> >> As I asked before, are you loading mod_php into the same Apache. That >> often causes shared library conflicts with different versions and/or >> use of non thread safe versions of libraries. >> >> If you can catch the process in that 5 minute window while daemon >> process is hung, you can use recipe at end of: >> >> >> http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Debugging_Crashes_With_GDB >> >> at work out where all the C threads are when it is stuck. This can >> indicate which C extension module might be the problem. >> >> Graham >> >> > I did get some segmentation faults earlier but somehow they stopped >> > appearing now. They were like this: >> > >> > [notice] seg fault or similar nasty error detected in the parent process >> > >> > Also, I get these messages: >> > >> > [error] server reached MaxClients setting, consider raising the >> > MaxClients >> > setting >> > >> > Prolly the last one is because all the processes get stucked and hence >> > the >> > server stops responding. >> > >> > I've changed the number of processes of WSGIDaemon now. Previously they >> > were >> > 1 and now they are 10. The config now looks like this: >> > >> > Here are my WSGI related lines: >> > >> > WSGIScriptAlias / /home/www/site.com/apache2/django.wsgi >> > WSGIDaemonProcess snr user=iwitness group=iwitness >> > processes=10 threads=10 display-name=apache-wsgi >> > WSGIProcessGroup snr >> > <Directory /home/www/site.com/apache2/django.wsgi> >> > Order allow,deny >> > Allow from all >> > </Directory> >> > >> > Sharjeel >> > >> > On Thu, Apr 8, 2010 at 2:36 PM, Graham Dumpleton >> > <[email protected]> wrote: >> >> >> >> On 8 April 2010 00:02, Sharjeel Qureshi <[email protected]> >> >> wrote: >> >> > I am having problems with my Django Based application. The server >> >> > stops responding occasionally, starts creating lot of processes until >> >> > it is restarted. Even when I shutdown apache, lot of apache processes >> >> > can be seen with "ps -aux | grep apache" command so I also have to do >> >> > "sudo killall -v apache2". >> >> > >> >> > My setup is on a 64-bit Ubuntu 8.04 machine. Main webserver is nginx >> >> > (which gives a 502 server timeout when it happens). It is connected >> >> > to >> >> > Apache 2.2 via reverse proxy. Apache is using MPM-Prefork and runs >> >> > the >> >> > mod_wsgi in daemon mode with a single process. The application is >> >> > using django 1.1 >> >> > >> >> > Here is my system configuration >> >> > >> >> > Ubuntu 6.04 >> >> > nginx for static content, reverse proxied with apache (nginx stays >> >> > stable and gives 502 server timeout) >> >> > Apache 2.2, mpm-prefork, maxclients=150 >> >> > Mod_wsgi 2.7, processes=1 >> >> > mod_python is NOT installed >> >> > django 1.1 >> >> > >> >> > I usually don't get more than 10 requests at each time but the >> >> > process >> >> > count becomes way higher than it should. >> >> > >> >> > I've changed the log level of both core apache as well as the virtual >> >> > host. In the errors log I've found following: >> >> > >> >> > a) [error] [client 127.0.0.1] mod_wsgi (pid=23988): Exception >> >> > occurred >> >> > processing WSGI script '/home/www/example.com/apache2/django.wsgi'. >> >> > b) [error] [client 127.0.0.1] IOError: failed to write data >> >> > c) [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: >> >> > writing data to the network >> >> > d) [error] [client 127.0.0.1] (4)Interrupted system call: mod_wsgi >> >> > (pid=24215): Unable to connect to WSGI daemon process 'snr' on '/var/ >> >> > run/apache2/wsgi.23609.0.1.sock' after multiple attempts. >> >> > e) lots of: [error] [client 127.0.0.1] Premature end of script >> >> > headers: django.wsgi >> >> > >> >> > I have similar setup running on three other instances but never had >> >> > problems on them strangely! >> >> > >> >> > Any ideas? >> >> >> >> Are you ever seeing any of the messages: >> >> >> >> Daemon process deadlock timer expired, stopping process ... >> >> >> >> Segmentation Fault >> >> >> >> When you see the message: >> >> >> >> Premature end of script headers >> >> >> >> are you seeing messages indicating that the daemon process has >> >> restarted? >> >> >> >> Ensure you check both main Apache error log and any virtual host >> >> specific error log for messages. >> >> >> >> At least while work this out, you can add option: >> >> >> >> inactivity-timeout=120 >> >> >> >> to WSGIDaemonProcess and should hopefully cause process to restart >> >> automatically after being stuck for 2 minutes. At least then if you >> >> can't get to it quickly yourself to restart Apache, it should fix >> >> itself up. >> >> >> >> BTW, do you have a specific need to run prefork MPM? If you don't, ie, >> >> not running mod_php, you would be better off with worker MPM. >> >> >> >> Also post the actual existing WSGIDaemonProcess/WSGIProcessGroup lines >> >> you are using now. >> >> >> >> Graham >> >> >> >> -- >> >> You received this message because you are subscribed to the Google >> >> Groups >> >> "modwsgi" group. >> >> To post to this group, send email to [email protected]. >> >> To unsubscribe from this group, send email to >> >> [email protected]. >> >> For more options, visit this group at >> >> http://groups.google.com/group/modwsgi?hl=en. >> >> >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "modwsgi" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group at >> > http://groups.google.com/group/modwsgi?hl=en. >> > >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/modwsgi?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/modwsgi?hl=en. > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
