Use gdb as discussed before to try and capture a stack trace of where problem occurs. Only way you can even start to work it out.
Also list what third party Python packages you are using which have a C extension component. Can do this by looking for any packages in your site-packages which have a .so file. Graham On 28 November 2011 01:11, caroline <[email protected]> wrote: > Hi > > What can i do for detecting which c extension module isn't thread > safe ? > > Caro > > > > On 26 nov, 23:21, Graham Dumpleton <[email protected]> wrote: >> You could be using a C extension module which isn't thread safe. >> >> BTW, you didn't need to force single thread to attach gdb to try and >> capture crash, but perhaps fortuitous you did as has narrowed down >> what the problem is. :-) >> >> Graham >> >> On 27 November 2011 14:55, caroline <[email protected]> wrote: >> >> >> >> >> >> >> >> > Hello Graham >> >> > I've made some more tests, using the debugging technique that you >> > wrote (with gdb), and when i put threads=1 the premature end of script >> > headers message doesn't appear anymore, so i couldn't see anything >> > relevant on gdb, Is this normal?. I repeatedly tested the application >> > without any values or with threads greater than 1 in the wsgi daemon >> > process definition (without using gdb of course) but the message >> > appeared again, so now i'm kind of confused. >> > Why could be the reason for the message doesn't appearing when >> > threads=1? >> >> > Thanks for your help >> >> > -- >> >> > On 23 nov, 18:01, Graham Dumpleton <[email protected]> wrote: >> >> Do you run any sort of monitoring system which kills processes >> >> automatically if memory exceeds some set amount? >> >> >> Only other thing can think of is that something is sending a single to >> >> daemon processes and killing them. This can't just be SIGINT or SIGHUP >> >> though as the daemon process in those cases would log something and it >> >> isn't that I can see from the parts of the virtual host log file you >> >> have posted. >> >> >> BTW, you might consider New Relic (www.newrelic.com) trial to see if >> >> it that sheds any light. Both the Python agent and server monitoring. >> >> >> Disclaimer. I work at New Relic and am writing the Python agent. The >> >> pro trial you get on initial sign up and before it drops down to free >> >> lite subscription still could be useful to help track things down, >> >> although if processes are dying suddenly then may not. >> >> >> Graham >> >> >> On 24 November 2011 09:49, caroline <[email protected]> wrote: >> >> >> > Hello Graham >> >> >> > On 22 nov, 17:20, Graham Dumpleton <[email protected]> wrote: >> >> >> On 23 November 2011 01:33, caroline <[email protected]> wrote: >> >> >> >> > Hi Graham >> >> >> >> > I've made the change you mentioned on the main apache configuration >> >> >> > file, but now the error message of 'premature end of script headers' >> >> >> > appears almost constantly. It appears like this in the vhost error >> >> >> > log: >> >> >> >> > [Tue Nov 22 09:21:19 2011] [error] [client 172.16.0.2] Premature end >> >> >> > of script headers: pinax.wsgi, referer: >> >> >> > http:///www.myvhost2.com/account/login/?next=/ >> >> >> > [Tue Nov 22 09:21:19 2011] [error] [client 172.16.0.2] Premature end >> >> >> > of script headers: pinax.wsgi, >> >> >> > referer:http://www.myvhost2.com/lms/courses/ >> >> >> > [Tue Nov 22 09:21:19 2011] [error] [client 172.16.0.2] Premature end >> >> >> > of script headers: pinax.wsgi, >> >> >> > referer:http://www.myvhost2.com/lms/courses/ >> >> >> > [Tue Nov 22 09:21:20 2011] [info] mod_wsgi (pid=25450): Attach >> >> >> > interpreter ''. >> >> >> > [Tue Nov 22 09:21:20 2011] [info] mod_wsgi (pid=25450): Adding '/var/ >> >> >> > quicklearn/env/lib/python2.6/site-packages' to path. >> >> >> > [Tue Nov 22 09:21:20 2011] [info] mod_wsgi (pid=25450): Adding '/var/ >> >> >> > quicklearn/trunk' to path. >> >> >> > [Tue Nov 22 09:21:21 2011] [info] [client 172.16.0.2] mod_wsgi >> >> >> > (pid=25450, process='quicklearn', application=''): Loading WSGI >> >> >> > script >> >> >> > '/var/quicklearn/trunk/deploy/pinax.wsgi'. >> >> >> >> Anything prior to this in same time frame? >> >> >> >> > And in the apache main error log at the same time : >> >> >> >> > [Tue Nov 22 09:20:31 2011] [info] mod_wsgi (pid=25149): Process >> >> >> > 'quicklearn' has died, restarting. >> >> >> >> Were there any messages in main Apache error log prior to this in same >> >> >> time frame. >> >> >> > In the apache error log file i only found messages like this: >> >> >> > [Wed Nov 23 17:01:59 2011] [info] mod_wsgi (pid=25954): Process >> >> > 'quicklearn' has died, restarting. >> >> > [Wed Nov 23 17:01:59 2011] [info] mod_wsgi (pid=14133): Starting >> >> > process 'quicklearn' with uid=48, gid=48 and threads=16. >> >> > [Wed Nov 23 17:02:09 2011] [debug] proxy_util.c(1818): proxy: grabbed >> >> > scoreboard slot 0 in child 14165 for worker proxy:reverse >> >> > [Wed Nov 23 17:02:09 2011] [debug] proxy_util.c(1837): proxy: worker >> >> > proxy:reverse already initialized >> >> > [Wed Nov 23 17:02:09 2011] [debug] proxy_util.c(1934): proxy: >> >> > initialized single connection worker 0 in child 14165 for (*) >> >> > [Wed Nov 23 17:02:44 2011] [debug] proxy_util.c(1818): proxy: grabbed >> >> > scoreboard slot 0 in child 14169 for worker proxy:reverse >> >> > [Wed Nov 23 17:02:44 2011] [debug] proxy_util.c(1837): proxy: worker >> >> > proxy:reverse already initialized >> >> > [Wed Nov 23 17:02:44 2011] [debug] proxy_util.c(1934): proxy: >> >> > initialized single connection worker 0 in child 14169 for (*) >> >> > [Wed Nov 23 17:03:38 2011] [debug] proxy_util.c(1818): proxy: grabbed >> >> > scoreboard slot 0 in child 14187 for worker proxy:reverse >> >> > [Wed Nov 23 17:03:38 2011] [debug] proxy_util.c(1837): proxy: worker >> >> > proxy:reverse already initialized >> >> > [Wed Nov 23 17:03:38 2011] [debug] proxy_util.c(1934): proxy: >> >> > initialized single connection worker 0 in child 14187 for (*) >> >> > [Wed Nov 23 17:04:20 2011] [info] mod_wsgi (pid=25953): Process >> >> > 'quicklearn' has died, restarting. >> >> > [Wed Nov 23 17:04:20 2011] [info] mod_wsgi (pid=14198): Starting >> >> > process 'quicklearn' with uid=48, gid=48 and threads=16. >> >> > [Wed Nov 23 17:04:23 2011] [debug] proxy_util.c(1818): proxy: grabbed >> >> > scoreboard slot 0 in child 14222 for worker proxy:reverse >> >> > [Wed Nov 23 17:04:23 2011] [debug] proxy_util.c(1837): proxy: worker >> >> > proxy:reverse already initialized >> >> > [Wed Nov 23 17:04:23 2011] [debug] proxy_util.c(1934): proxy: >> >> > initialized single connection worker 0 in child 14222 for (*) >> >> > [Wed Nov 23 17:04:48 2011] [debug] proxy_util.c(1818): proxy: grabbed >> >> > scoreboard slot 0 in child 14227 for worker proxy:reverse >> >> > [Wed Nov 23 17:04:48 2011] [debug] proxy_util.c(1837): proxy: worker >> >> > proxy:reverse already initialized >> >> > [Wed Nov 23 17:04:48 2011] [debug] proxy_util.c(1934): proxy: >> >> > initialized single connection worker 0 in child 14227 for (*) >> >> > [Wed Nov 23 17:04:51 2011] [debug] proxy_util.c(1818): proxy: grabbed >> >> > scoreboard slot 0 in child 14228 for worker proxy:reverse >> >> > [Wed Nov 23 17:04:51 2011] [debug] proxy_util.c(1837): proxy: worker >> >> > proxy:reverse already initialized >> >> > [Wed Nov 23 17:04:51 2011] [debug] proxy_util.c(1934): proxy: >> >> > initialized single connection worker 0 in child 14228 for (*) >> >> > [Wed Nov 23 17:05:05 2011] [info] mod_wsgi (pid=14198): Process >> >> > 'quicklearn' has died, restarting. >> >> > [Wed Nov 23 17:05:05 2011] [info] mod_wsgi (pid=14238): Starting >> >> > process 'quicklearn' with uid=48, gid=48 and threads=16. >> >> >> >> It may not be segmentation fault. It is generally though a message >> >> >> which doesn't have the date/time prefix as output at a lower level >> >> >> than Apache logging API. Provide the preceding messages so I can look >> >> >> through. >> >> >> > I've searched for messages like this on logs but i haven't found >> >> > anything like you mention. >> >> >> >> If you operating system has a crash logs directory you might look in >> >> >> there for evidence of crashing apache/httpd processes. >> >> >> >> If the problem is now more frequent, try and work out if it is a >> >> >> specific URL. Look in Apache access logs to see if 500 error status >> >> >> for same URL a lot of the time. In other words, try and reliably >> >> >> duplicate it. >> >> >> >> Being able to duplicate it, you could then attach gdb to a mod_wsgi >> >> >> daemon process and try and catch the crash. Details on how to do that >> >> >> can be found in: >> >> >> >>http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Debugging_C... >> >> >> > I'm going to apply this technique, and i'll inform you about its >> >> > results. >> >> >> >> > [Tue Nov 22 09:20:31 2011] [info] mod_wsgi (pid=25413): Starting >> >> >> > process 'quicklearn' with uid=48, gid=48 and threads=16. >> >> >> > [Tue Nov 22 09:21:20 2011] [info] mod_wsgi (pid=25145): Process >> >> >> > 'quicklearn' has died, restarting. >> >> >> > [Tue Nov 22 09:21:20 2011] [info] mod_wsgi (pid=25450): Starting >> >> >> > process 'quicklearn' with uid=48, gid=48 and threads=16. >> >> >> > [Tue Nov 22 09:21:32 2011] [debug] proxy_util.c(1818): proxy: grabbed >> >> >> > scoreboard slot 0 in child 25475 for worker proxy:reverse >> >> >> > [Tue Nov 22 09:21:32 2011] [debug] proxy_util.c(1837): proxy: worker >> >> >> > proxy:reverse already initialized >> >> >> > [Tue Nov 22 09:21:32 2011] [debug] proxy_util.c(1934): proxy: >> >> >> > initialized single connection worker 0 in child 25475 for (*) >> >> >> > [Tue Nov 22 09:22:31 2011] [debug] proxy_util.c(1818): proxy: grabbed >> >> >> > scoreboard slot 0 in child 25493 for worker proxy:reverse >> >> >> > [Tue Nov 22 09:22:31 2011] [debug] proxy_util.c(1837): proxy: worker >> >> >> > proxy:reverse already initialized >> >> >> > [Tue Nov 22 09:22:31 2011] [debug] proxy_util.c(1934): proxy: >> >> >> > initialized single connection worker 0 in child 25493 for (*) >> >> >> >> > I have this on the main apache configuration file: >> >> >> >> > WSGILazyInitialization Off >> >> >> > WSGIRestrictEmbedded On >> >> >> >> BTW, you sure you don't mean: >> >> >> >> WSGILazyInitialization On >> >> >> WSGIRestrictEmbedded On >> >> >> >> You generally wouldn't want to turn off lazy initialisation as Python >> >> >> interpreter leaks memory into Apache parent process on restarts if >> >> >> lazy initialisation is turned off. >> >> >> >> Graham >> >> >> > Thanks about that! i changed this too >> >> >> >> > WSGIApplicationGroup %{GLOBAL} >> >> >> >> > But it's not working. And i've searched for the segmentation fault >> >> >> > message but it doesn't appear anywhere. >> >> >> >> > What more could i do? >> >> >> >> > Thanks >> >> >> >> > On 21 nov, 18:52, Graham Dumpleton <[email protected]> >> >> >> > wrote: >> >> >> >> On 22 November 2011 09:35, caroline... >> >> leer más » > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/modwsgi?hl=en. > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
