On Sun, Mar 10, 2013 at 8:26 PM, andrew mcelroy <[email protected]> wrote: > On Sun, Mar 10, 2013 at 8:18 PM, Toth, Csaba <[email protected]> > wrote: >> Hey Linux Gurus, >> >> I wonder about your advices how I can solve this. >> Our lab's web server at Vandy crashes every Sunday at 7:41am exactly. >> I cannot track down the culprit. It's an Ubuntu Server Edition 12.04.1 LTS >> Precise. >> Apache has php5 and even Drupal. We don't use Drupal and very little php. >> >> What I suspect that it's some scheduled task doing the harm. >> The Anachron weekly starts at 7:30am. After that it start the cron.weekly >> jobs with 10 minutes separation. Cron.weekly: >> "0anacron >> apt-xapian-index > > Is it possible that apt-xapian-index is causing a cpu spike and > somehow causing apache to have that behavior? > Do you have any logs on xapian? > > http://forums.linuxmint.com/viewtopic.php?f=90&t=82985 > > Andrew McElroy
related: There was an old bug where the mod_python would deadlock due to xapian. Andrew McElroy > > >> debsums >> man-db >> sysklogd" >> >> 7:41 crash means that it can be the second thing it starts. I don't know if >> 0anacron counts or not. Debsums: this is a package with which you can >> identify damaged package. (The server had HDD hardware failure during the >> summer. That was degrading for a while, and we had logical file system >> issues too. I was down at the Medical Center DataCenter and I had to issue >> more fscks to recover the filesystem fully. I'm even happy that I could save >> the RAID volume, since it fell apart for sometime too.) >> I went ahead and started all scripts in cron.weekly manually from a root >> command prompt, but nothing caused a crash. More info about the crash: I >> cannot see anything in the Apache access logs for 7:41am, the last log is >> for 7:37am. >> >> Error log: >> >> "[Sun Mar 10 01:29:09 2013] [error] [client 108.222.66.23] File does not >> exist: /var/www/www.hiplab.org/favicon.ico >> [Sun Mar 10 01:50:02 2013] [error] [client 66.249.76.209] File does not >> exist: /var/www/www.hiplab.org/people/malin/BMIF380 >> [Sun Mar 10 03:36:04 2013] [error] [client 41.196.103.228] File does not >> exist: /var/www/www.hiplab.org/favicon.ico >> [Sun Mar 10 03:36:08 2013] [error] [client 41.196.103.228] File does not >> exist: /var/www/www.hiplab.org/favicon.ico >> [Sun Mar 10 03:36:40 2013] [error] [client 41.196.103.228] File does not >> exist: /var/www/www.hiplab.org/favicon.ico >> [Sun Mar 10 03:52:46 2013] [error] [client 199.30.20.73] File does not >> exist: /var/www/www.hiplab.org/people/weiyixia/icon.jpg >> [Sun Mar 10 04:46:58 2013] [error] [client 78.162.114.93] File does not >> exist: /var/www/www.hiplab.org/favicon.ico >> [Sun Mar 10 06:52:30 2013] [error] [client 69.163.180.32] File does not >> exist: /var/www/www.hiplab.org/wp-admin >> [Sun Mar 10 07:05:50 2013] [error] [client 69.163.187.202] File does not >> exist: /var/www/www.hiplab.org/wordpress >> [Sun Mar 10 07:26:48 2013] [error] [client 69.163.148.155] File does not >> exist: /var/www/www.hiplab.org/wordpress >> [Sun Mar 10 07:41:47 2013] [notice] Graceful restart requested, doing >> restart >> [Sun Mar 10 07:41:47 2013] [alert] (4)Interrupted system call: FastCGI: >> read() from pipe failed (0) >> [Sun Mar 10 07:41:47 2013] [alert] (4)Interrupted system call: FastCGI: the >> PM is shutting down, Apache seems to have disappeared - bye >> *** glibc detected *** /usr/sbin/apache2: corrupted double-linked list: >> 0x00007fda105bae40 *** >> ======= Backtrace: ========= >> /lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7fda0e8b1b96] >> /lib/x86_64-linux-gnu/libc.so.6(+0x7eec8)[0x7fda0e8b1ec8] >> /lib/x86_64-linux-gnu/libc.so.6(+0x80436)[0x7fda0e8b3436] >> /lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x75)[0x7fda0e8b5fb5] >> /lib/x86_64-linux-gnu/libc.so.6(+0xbb50b)[0x7fda0e8ee50b] >> /usr/lib/libapr-1.so.0(apr_dir_open+0x29)[0x7fda0ee24e76] >> /usr/sbin/apache2(ap_process_resource_config+0xed)[0x7fda0f71030d] >> /usr/sbin/apache2(+0x35aaa)[0x7fda0f702aaa] >> /usr/sbin/apache2(+0x40741)[0x7fda0f70d741] >> /usr/sbin/apache2(+0x42304)[0x7fda0f70f304] >> /usr/sbin/apache2(ap_build_config+0x9c)[0x7fda0f70f8cc] >> /usr/sbin/apache2(+0x42d01)[0x7fda0f70fd01] >> /usr/sbin/apache2(ap_process_resource_config+0x209)[0x7fda0f710429] >> /usr/sbin/apache2(ap_read_config+0x2df)[0x7fda0f71104f] >> /usr/sbin/apache2(main+0xbb6)[0x7fda0f6f93d6] >> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7fda0e85476d] >> /usr/sbin/apache2(+0x2c559)[0x7fda0f6f9559] >> ======= Memory map: ========" >> >> In the beginning you can see nice tries from bots trying to grab >> vulnerabilities: mofo bots tirelessly try to seek for admin consoles. Then >> at 07:41:47 restart requested, and apache segfaults. Crash dump is generated >> also. >> >> One question: what else can start regularly weekly which is not anacron? >> Any other advice is welcome, I am clueless at this moment. >> >> Thanks, >> Csaba >> >> -- >> -- >> You received this message because you are subscribed to the Google Groups >> "NLUG" group. >> To post to this group, send email to [email protected] >> To unsubscribe from this group, send email to >> [email protected] >> For more options, visit this group at >> http://groups.google.com/group/nlug-talk?hl=en >> >> --- >> You received this message because you are subscribed to the Google Groups >> "NLUG" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> -- -- You received this message because you are subscribed to the Google Groups "NLUG" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nlug-talk?hl=en --- You received this message because you are subscribed to the Google Groups "NLUG" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
