You could be using a C extension module which isn't thread safe. BTW, you didn't need to force single thread to attach gdb to try and capture crash, but perhaps fortuitous you did as has narrowed down what the problem is. :-)
Graham On 27 November 2011 14:55, caroline <[email protected]> wrote: > Hello Graham > > I've made some more tests, using the debugging technique that you > wrote (with gdb), and when i put threads=1 the premature end of script > headers message doesn't appear anymore, so i couldn't see anything > relevant on gdb, Is this normal?. I repeatedly tested the application > without any values or with threads greater than 1 in the wsgi daemon > process definition (without using gdb of course) but the message > appeared again, so now i'm kind of confused. > Why could be the reason for the message doesn't appearing when > threads=1? > > > Thanks for your help > > > -- > > > > On 23 nov, 18:01, Graham Dumpleton <[email protected]> wrote: >> Do you run any sort of monitoring system which kills processes >> automatically if memory exceeds some set amount? >> >> Only other thing can think of is that something is sending a single to >> daemon processes and killing them. This can't just be SIGINT or SIGHUP >> though as the daemon process in those cases would log something and it >> isn't that I can see from the parts of the virtual host log file you >> have posted. >> >> BTW, you might consider New Relic (www.newrelic.com) trial to see if >> it that sheds any light. Both the Python agent and server monitoring. >> >> Disclaimer. I work at New Relic and am writing the Python agent. The >> pro trial you get on initial sign up and before it drops down to free >> lite subscription still could be useful to help track things down, >> although if processes are dying suddenly then may not. >> >> Graham >> >> On 24 November 2011 09:49, caroline <[email protected]> wrote: >> >> >> >> >> >> >> >> > Hello Graham >> >> > On 22 nov, 17:20, Graham Dumpleton <[email protected]> wrote: >> >> On 23 November 2011 01:33, caroline <[email protected]> wrote: >> >> >> > Hi Graham >> >> >> > I've made the change you mentioned on the main apache configuration >> >> > file, but now the error message of 'premature end of script headers' >> >> > appears almost constantly. It appears like this in the vhost error >> >> > log: >> >> >> > [Tue Nov 22 09:21:19 2011] [error] [client 172.16.0.2] Premature end >> >> > of script headers: pinax.wsgi, referer: >> >> > http:///www.myvhost2.com/account/login/?next=/ >> >> > [Tue Nov 22 09:21:19 2011] [error] [client 172.16.0.2] Premature end >> >> > of script headers: pinax.wsgi, >> >> > referer:http://www.myvhost2.com/lms/courses/ >> >> > [Tue Nov 22 09:21:19 2011] [error] [client 172.16.0.2] Premature end >> >> > of script headers: pinax.wsgi, >> >> > referer:http://www.myvhost2.com/lms/courses/ >> >> > [Tue Nov 22 09:21:20 2011] [info] mod_wsgi (pid=25450): Attach >> >> > interpreter ''. >> >> > [Tue Nov 22 09:21:20 2011] [info] mod_wsgi (pid=25450): Adding '/var/ >> >> > quicklearn/env/lib/python2.6/site-packages' to path. >> >> > [Tue Nov 22 09:21:20 2011] [info] mod_wsgi (pid=25450): Adding '/var/ >> >> > quicklearn/trunk' to path. >> >> > [Tue Nov 22 09:21:21 2011] [info] [client 172.16.0.2] mod_wsgi >> >> > (pid=25450, process='quicklearn', application=''): Loading WSGI script >> >> > '/var/quicklearn/trunk/deploy/pinax.wsgi'. >> >> >> Anything prior to this in same time frame? >> >> >> > And in the apache main error log at the same time : >> >> >> > [Tue Nov 22 09:20:31 2011] [info] mod_wsgi (pid=25149): Process >> >> > 'quicklearn' has died, restarting. >> >> >> Were there any messages in main Apache error log prior to this in same >> >> time frame. >> >> > In the apache error log file i only found messages like this: >> >> > [Wed Nov 23 17:01:59 2011] [info] mod_wsgi (pid=25954): Process >> > 'quicklearn' has died, restarting. >> > [Wed Nov 23 17:01:59 2011] [info] mod_wsgi (pid=14133): Starting >> > process 'quicklearn' with uid=48, gid=48 and threads=16. >> > [Wed Nov 23 17:02:09 2011] [debug] proxy_util.c(1818): proxy: grabbed >> > scoreboard slot 0 in child 14165 for worker proxy:reverse >> > [Wed Nov 23 17:02:09 2011] [debug] proxy_util.c(1837): proxy: worker >> > proxy:reverse already initialized >> > [Wed Nov 23 17:02:09 2011] [debug] proxy_util.c(1934): proxy: >> > initialized single connection worker 0 in child 14165 for (*) >> > [Wed Nov 23 17:02:44 2011] [debug] proxy_util.c(1818): proxy: grabbed >> > scoreboard slot 0 in child 14169 for worker proxy:reverse >> > [Wed Nov 23 17:02:44 2011] [debug] proxy_util.c(1837): proxy: worker >> > proxy:reverse already initialized >> > [Wed Nov 23 17:02:44 2011] [debug] proxy_util.c(1934): proxy: >> > initialized single connection worker 0 in child 14169 for (*) >> > [Wed Nov 23 17:03:38 2011] [debug] proxy_util.c(1818): proxy: grabbed >> > scoreboard slot 0 in child 14187 for worker proxy:reverse >> > [Wed Nov 23 17:03:38 2011] [debug] proxy_util.c(1837): proxy: worker >> > proxy:reverse already initialized >> > [Wed Nov 23 17:03:38 2011] [debug] proxy_util.c(1934): proxy: >> > initialized single connection worker 0 in child 14187 for (*) >> > [Wed Nov 23 17:04:20 2011] [info] mod_wsgi (pid=25953): Process >> > 'quicklearn' has died, restarting. >> > [Wed Nov 23 17:04:20 2011] [info] mod_wsgi (pid=14198): Starting >> > process 'quicklearn' with uid=48, gid=48 and threads=16. >> > [Wed Nov 23 17:04:23 2011] [debug] proxy_util.c(1818): proxy: grabbed >> > scoreboard slot 0 in child 14222 for worker proxy:reverse >> > [Wed Nov 23 17:04:23 2011] [debug] proxy_util.c(1837): proxy: worker >> > proxy:reverse already initialized >> > [Wed Nov 23 17:04:23 2011] [debug] proxy_util.c(1934): proxy: >> > initialized single connection worker 0 in child 14222 for (*) >> > [Wed Nov 23 17:04:48 2011] [debug] proxy_util.c(1818): proxy: grabbed >> > scoreboard slot 0 in child 14227 for worker proxy:reverse >> > [Wed Nov 23 17:04:48 2011] [debug] proxy_util.c(1837): proxy: worker >> > proxy:reverse already initialized >> > [Wed Nov 23 17:04:48 2011] [debug] proxy_util.c(1934): proxy: >> > initialized single connection worker 0 in child 14227 for (*) >> > [Wed Nov 23 17:04:51 2011] [debug] proxy_util.c(1818): proxy: grabbed >> > scoreboard slot 0 in child 14228 for worker proxy:reverse >> > [Wed Nov 23 17:04:51 2011] [debug] proxy_util.c(1837): proxy: worker >> > proxy:reverse already initialized >> > [Wed Nov 23 17:04:51 2011] [debug] proxy_util.c(1934): proxy: >> > initialized single connection worker 0 in child 14228 for (*) >> > [Wed Nov 23 17:05:05 2011] [info] mod_wsgi (pid=14198): Process >> > 'quicklearn' has died, restarting. >> > [Wed Nov 23 17:05:05 2011] [info] mod_wsgi (pid=14238): Starting >> > process 'quicklearn' with uid=48, gid=48 and threads=16. >> >> >> It may not be segmentation fault. It is generally though a message >> >> which doesn't have the date/time prefix as output at a lower level >> >> than Apache logging API. Provide the preceding messages so I can look >> >> through. >> >> > I've searched for messages like this on logs but i haven't found >> > anything like you mention. >> >> >> If you operating system has a crash logs directory you might look in >> >> there for evidence of crashing apache/httpd processes. >> >> >> If the problem is now more frequent, try and work out if it is a >> >> specific URL. Look in Apache access logs to see if 500 error status >> >> for same URL a lot of the time. In other words, try and reliably >> >> duplicate it. >> >> >> Being able to duplicate it, you could then attach gdb to a mod_wsgi >> >> daemon process and try and catch the crash. Details on how to do that >> >> can be found in: >> >> >>http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Debugging_C... >> >> > I'm going to apply this technique, and i'll inform you about its >> > results. >> >> >> > [Tue Nov 22 09:20:31 2011] [info] mod_wsgi (pid=25413): Starting >> >> > process 'quicklearn' with uid=48, gid=48 and threads=16. >> >> > [Tue Nov 22 09:21:20 2011] [info] mod_wsgi (pid=25145): Process >> >> > 'quicklearn' has died, restarting. >> >> > [Tue Nov 22 09:21:20 2011] [info] mod_wsgi (pid=25450): Starting >> >> > process 'quicklearn' with uid=48, gid=48 and threads=16. >> >> > [Tue Nov 22 09:21:32 2011] [debug] proxy_util.c(1818): proxy: grabbed >> >> > scoreboard slot 0 in child 25475 for worker proxy:reverse >> >> > [Tue Nov 22 09:21:32 2011] [debug] proxy_util.c(1837): proxy: worker >> >> > proxy:reverse already initialized >> >> > [Tue Nov 22 09:21:32 2011] [debug] proxy_util.c(1934): proxy: >> >> > initialized single connection worker 0 in child 25475 for (*) >> >> > [Tue Nov 22 09:22:31 2011] [debug] proxy_util.c(1818): proxy: grabbed >> >> > scoreboard slot 0 in child 25493 for worker proxy:reverse >> >> > [Tue Nov 22 09:22:31 2011] [debug] proxy_util.c(1837): proxy: worker >> >> > proxy:reverse already initialized >> >> > [Tue Nov 22 09:22:31 2011] [debug] proxy_util.c(1934): proxy: >> >> > initialized single connection worker 0 in child 25493 for (*) >> >> >> > I have this on the main apache configuration file: >> >> >> > WSGILazyInitialization Off >> >> > WSGIRestrictEmbedded On >> >> >> BTW, you sure you don't mean: >> >> >> WSGILazyInitialization On >> >> WSGIRestrictEmbedded On >> >> >> You generally wouldn't want to turn off lazy initialisation as Python >> >> interpreter leaks memory into Apache parent process on restarts if >> >> lazy initialisation is turned off. >> >> >> Graham >> >> > Thanks about that! i changed this too >> >> >> > WSGIApplicationGroup %{GLOBAL} >> >> >> > But it's not working. And i've searched for the segmentation fault >> >> > message but it doesn't appear anywhere. >> >> >> > What more could i do? >> >> >> > Thanks >> >> >> > On 21 nov, 18:52, Graham Dumpleton <[email protected]> wrote: >> >> >> On 22 November 2011 09:35, caroline <[email protected]> wrote: >> >> >> >> > Hello >> >> >> >> > Thanks for your quick response. >> >> >> >> > This message appears randomly, i'm not doing apache restart. >> >> >> >> Have a very good look through your main Apache error log (not virtual >> >> >> host) for messages like 'segmentation fault'. >> >> >> >> For it to happen randomly, would indicate that the mod_wsgi daemon >> >> >> process is crashing. >> >> >> >> If this is the only Python web application being hosted add to Apache >> >> >> configuration: >> >> >> >> WSGIApplicationGroup %{GLOBAL} >> >> >> >> This can work around issues with third party C extension modules for >> >> >> Python which aren't written to work properly in Python sub >> >> >> interpreters within a process. The directive will force use of main >> >> >> Python interpreter within process. >> >> >> >> Graham >> >> >> >> > I have >> >> >> > been doing stress tests on this application and this is the only way >> >> >> > that i can easily reproduce this issue on this server. >> >> >> > First i thougth that it was a problem with concurrent users, but when >> >> >> > there was only one user logged in , the message appeared too. >> >> >> >> > -- >> >> >> >> > Caro >> >> >> >> > On 21 nov, 15:01, Graham Dumpleton <[email protected]> >> >> >> > wrote: >> >> >> >> Can you clarify whether the message about premature end of script >> >> >> >> headers >> >> >> >> is on every request or only when you perform an apache restart? If >> >> >> >> on >> >> >> >> restart >> >> ... >> >> leer más » > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/modwsgi?hl=en. > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
