Okay. So this isn't an atexit() callback but global C++ object destructors kicking in from the automatic execution of finaliser sections on the object files.
Same issue applies though in part. It looks like the page speed module could be making some assumption that certain data will always be initialised by the time the process is terminated, but possibly because Apache module child init handlers are not called for the page speed module in the mod_wsgi daemon processes, then that data isn't initialised and as a result it crashes. When this happens though it is usual to see a NULL pointer dereference or low memory access due to relative reference to NULL pointer. I can't see an obvious case of that, but is hard to tell what the module is doing. Another problem with this thought is that since the page speed module doesn't get to do anything at all in the mod_wsgi daemon mode process, then can't see how this issue wouldn't also arise in the Apache parent process unless the fact that the module might be unloaded from memory by Apache first before shutdown (can't remember) might mean that global C++ destructors aren't called in that case. Now one could argue that if this is happening that the page speed module is being sloppy, but at the same time, under normal circumstances an Apache module would never need to contend with possibility that something like the Apache child init handler might not be called in a child process. That is an oddity caused by mod_wsgi daemon mode. Anyway, all can do right now is confirm whether it is the page speed module by disabling that module temporarily. Will then need to work out what to do and perhaps raise issue with page speed module authors if that is where it is arising and see if they want to say not their problem since mod_wsgi does weird stuff. :-) Graham On 13/05/2014, at 9:51 AM, Alex Wu <[email protected]> wrote: > Here is one example: > > warning: Can't read pathname for load map: Input/output error. > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > Core was generated by `(wsgi:dataplane) -D Dataplane -D > pagespeed -D fwd_proxy -D DAT'. > Program terminated with signal 11, Segmentation fault. > #0 0x00007fb46da18e4a in ?? () from /usr/lib/libpython2.7.so.1.0 > (gdb) info threads > Id Target Id Frame > 5 Thread 0x7fb458fe9700 (LWP 25847) 0x00007fb4777566e0 in sigprocmask () > from /lib/x86_64-linux-gnu/libc.so.6 > 4 Thread 0x7fb4785bb740 (LWP 22886) 0x00007fb46cadd678 in (anonymous > namespace)::scribble (ptr=0x7fb478f13a38, size=34008, > scribble_word=-559038737) > at pagespeed/kernel/base/mem_debug.cc:81 > 3 Thread 0x7fb469870700 (LWP 22895) 0x00007fb477814a93 in epoll_wait () > from /lib/x86_64-linux-gnu/libc.so.6 > 2 Thread 0x7fb46aa76700 (LWP 22893) 0x00007fb47780d763 in select () from > /lib/x86_64-linux-gnu/libc.so.6 > * 1 Thread 0x7fb46a071700 (LWP 22894) 0x00007fb46da18e4a in ?? () from > /usr/lib/libpython2.7.so.1.0 > (gdb) thread 4 > [Switching to thread 4 (Thread 0x7fb4785bb740 (LWP 22886))] > #0 0x00007fb46cadd678 in (anonymous namespace)::scribble > (ptr=0x7fb478f13a38, size=34008, scribble_word=-559038737) at > pagespeed/kernel/base/mem_debug.cc:81 > 81 pagespeed/kernel/base/mem_debug.cc: No such file or directory. > (gdb) bt > #0 0x00007fb46cadd678 in (anonymous namespace)::scribble > (ptr=0x7fb478f13a38, size=34008, scribble_word=-559038737) at > pagespeed/kernel/base/mem_debug.cc:81 > #1 0x00007fb46cadd827 in (anonymous namespace)::debug_free > (ptr=0x7fb478f13a38) at pagespeed/kernel/base/mem_debug.cc:100 > #2 0x00007fb46cadd9f9 in operator delete[] (ptr=0x7fb478f13a38) at > pagespeed/kernel/base/mem_debug.cc:142 > #3 0x00007fb46ce2256e in re2::Prog::~Prog (this=0x7fb478c260e8, > __in_chrg=<optimized out>) at third_party/re2/src/re2/prog.cc:123 > #4 0x00007fb46cdf5402 in re2::RE2::~RE2 (this=0x7fb478ff3dd8, > __in_chrg=<optimized out>) at third_party/re2/src/re2/re2.cc:272 > #5 0x00007fb46d1033af in > pagespeed::js::JsTokenizerPatterns::~JsTokenizerPatterns > (this=0x7fb478ff3dd8, __in_chrg=<optimized out>) > at pagespeed/kernel/js/js_tokenizer.cc:1096 > #6 0x00007fb46cf9f00c in > base::DefaultDeleter<pagespeed::js::JsTokenizerPatterns>::operator() > (this=0x7fb46d6a6fe8, ptr=0x7fb478ff3dd8) > at third_party/chromium/src/base/memory/scoped_ptr.h:137 > #7 0x00007fb46cf9efc2 in > base::internal::scoped_ptr_impl<pagespeed::js::JsTokenizerPatterns, > base::DefaultDeleter<pagespeed::js::JsTokenizerPatterns> >::~scoped_ptr_impl > (this=0x7fb46d6a6fe8, __in_chrg=<optimized out>) at > third_party/chromium/src/base/memory/scoped_ptr.h:220 > #8 0x00007fb46cf9ef6c in scoped_ptr<pagespeed::js::JsTokenizerPatterns, > base::DefaultDeleter<pagespeed::js::JsTokenizerPatterns> >::~scoped_ptr > (this=0x7fb46d6a6fe8, > __in_chrg=<optimized out>) at > third_party/chromium/src/base/memory/scoped_ptr.h:310 > #9 0x00007fb46cf9ef33 in net_instaweb::ProcessContext::~ProcessContext > (this=0x7fb46d6a6fe8, __in_chrg=<optimized out>) at > net/instaweb/rewriter/process_context.cc:54 > #10 0x00007fb46cad3969 in net_instaweb::(anonymous > namespace)::ApacheProcessContext::~ApacheProcessContext (this=0x7fb46d6a6fe0, > __in_chrg=<optimized out>) > at net/instaweb/apache/mod_instaweb.cc:313 > #11 0x00007fb47775b901 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 > #12 0x00007fb47775b985 in exit () from /lib/x86_64-linux-gnu/libc.so.6 > #13 0x00007fb46dddfd96 in wsgi_start_process (p=<optimized out>, > daemon=<optimized out>) at mod_wsgi.c:11969 > #14 0x00007fb46dde1344 in wsgi_start_daemons (p=0x7fb478bac138) at > mod_wsgi.c:12166 > #15 wsgi_hook_init (pconf=0x7fb478bac138, ptemp=<optimized out>, > plog=<optimized out>, s=<optimized out>) at mod_wsgi.c:13737 > #16 0x00007fb478633113 in ap_run_post_config (pconf=0x7fb478bac138, > plog=0x7fb478bd9378, ptemp=0x7fb478bd7348, s=0x7fb478bd5538) at config.c:106 > #17 0x00007fb478608993 in main (argc=15, argv=0x7fff2ee8cfd8) at main.c:765 > On Monday, May 12, 2014 4:07:35 PM UTC-7, Graham Dumpleton wrote: > Can you point out to me where in the Apache 2.4 code base it calls atexit() > to register anything on process shutdown? > > Neither Apache nor the underlying APR/APU libraries that it uses rely on > atexit() to have anything triggered on process shutdown that I know of and I > cannot find anything in the code I have handy for those which uses atexit() > in such a generic way. > > Normally Apache relies on cleanup actions attached to deletion of memory > pools and not atexit(). Thus it requires orderly Apache process shutdown and > for memory pools to be destroyed for actions to be performed on process > shutdown. The destruction of memory pools is not triggered via atexit(). > > Do you also have a more extensive stack trace that that one line so I can see > in what actual code the crash occurs? That may give me more clues. > > Graham > > On 13/05/2014, at 8:58 AM, Alex Wu <[email protected]> wrote: > >> we do not specifically add hook to atexit. It is called/triggered by apache >> frame work when a module is written within the apache 2.4 frame work. Also, >> mod_pagespeed used scoped point on their server context, it triggers auto >> clean once exit is called and library is unloaded. >> >> Alex >> >> >> >> On Monday, May 12, 2014 3:40:26 PM UTC-7, Graham Dumpleton wrote: >> If your own Apache modules are using atexit() to perform cleanup on process >> exit, rather than Apache's own mechanisms for performing cleanup actions >> when the pool the module uses is cleaned up, then the atexit() callback will >> have to take into consideration that under mod_wsgi when using daemon mode, >> that the Apache module child init handler will not be called in the daemon >> process for your Apache module. Thus the callback should check whether >> global data pointers are in fact non NULL before trying to do things with >> them. >> >> Can you confirm you are using atexit() callbacks in C code with your Apache >> modules and explain at what point you are registering the callback with >> atexit()? >> >> Is there a specific reason you are using atexit() callbacks rather than >> doing the normal thing of in the Apache module child init handler >> registering a cleanup callback on the memory pool given to the Apache module >> on child init and relying on that being triggered by Apache when shutting >> things down? >> >> Graham >> >> On 13/05/2014, at 8:23 AM, Alex Wu <[email protected]> wrote: >> >>> some are our own, one is mod_pagespeed. We use python 2.7.3 with apache >>> 2.4.7 in MPM mode. The segmentation fault is cleanup routine of each >>> modules other than mod_wsgi after exit call. >>> >>> Alex >>> >>> >>> On Monday, May 12, 2014 1:50:35 PM UTC-7, Graham Dumpleton wrote: >>> On 13/05/2014, at 4:40 AM, Alex Wu <[email protected]> wrote: >>> >>> > We have observed various segmentation fault caused by exit call from >>> > mod_wsgi 3.5: >>> > >>> > #20 0x00007f9490a94d96 in wsgi_start_process (p=<optimized out>, >>> > daemon=<optimized out>) at mod_wsgi.c:11969 >>> > >>> > The exit call triggers cleanup from other modules, that cleanup caused >>> > segmentation fault, >>> >>> What version of Apache and Python are you using? >>> >>> What other non standard Apache modules are you using? For example, is PHP >>> being used in the same Apache instance? >>> >>> Graham >>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/modwsgi. >>> For more options, visit https://groups.google.com/d/optout. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/modwsgi. >> For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/modwsgi. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
