Hey Linux Gurus, I wonder about your advices how I can solve this. Our lab's web server at Vandy crashes every Sunday at 7:41am exactly. I cannot track down the culprit. It's an Ubuntu Server Edition 12.04.1 LTS Precise. Apache has php5 and even Drupal. We don't use Drupal and very little php.
What I suspect that it's some scheduled task doing the harm. The Anachron weekly starts at 7:30am. After that it start the cron.weekly jobs with 10 minutes separation. Cron.weekly: "0anacron apt-xapian-index debsums man-db sysklogd" 7:41 crash means that it can be the second thing it starts. I don't know if 0anacron counts or not. Debsums: this is a package with which you can identify damaged package. (The server had HDD hardware failure during the summer. That was degrading for a while, and we had logical file system issues too. I was down at the Medical Center DataCenter and I had to issue more fscks to recover the filesystem fully. I'm even happy that I could save the RAID volume, since it fell apart for sometime too.) I went ahead and started all scripts in cron.weekly manually from a root command prompt, but nothing caused a crash. More info about the crash: I cannot see anything in the Apache access logs for 7:41am, the last log is for 7:37am. Error log: "[Sun Mar 10 01:29:09 2013] [error] [client 108.222.66.23] File does not exist: /var/www/www.hiplab.org/favicon.ico [Sun Mar 10 01:50:02 2013] [error] [client 66.249.76.209] File does not exist: /var/www/www.hiplab.org/people/malin/BMIF380 [Sun Mar 10 03:36:04 2013] [error] [client 41.196.103.228] File does not exist: /var/www/www.hiplab.org/favicon.ico [Sun Mar 10 03:36:08 2013] [error] [client 41.196.103.228] File does not exist: /var/www/www.hiplab.org/favicon.ico [Sun Mar 10 03:36:40 2013] [error] [client 41.196.103.228] File does not exist: /var/www/www.hiplab.org/favicon.ico [Sun Mar 10 03:52:46 2013] [error] [client 199.30.20.73] File does not exist: /var/www/www.hiplab.org/people/weiyixia/icon.jpg [Sun Mar 10 04:46:58 2013] [error] [client 78.162.114.93] File does not exist: /var/www/www.hiplab.org/favicon.ico [Sun Mar 10 06:52:30 2013] [error] [client 69.163.180.32] File does not exist: /var/www/www.hiplab.org/wp-admin [Sun Mar 10 07:05:50 2013] [error] [client 69.163.187.202] File does not exist: /var/www/www.hiplab.org/wordpress [Sun Mar 10 07:26:48 2013] [error] [client 69.163.148.155] File does not exist: /var/www/www.hiplab.org/wordpress [Sun Mar 10 07:41:47 2013] [notice] Graceful restart requested, doing restart [Sun Mar 10 07:41:47 2013] [alert] (4)Interrupted system call: FastCGI: read() from pipe failed (0) [Sun Mar 10 07:41:47 2013] [alert] (4)Interrupted system call: FastCGI: the PM is shutting down, Apache seems to have disappeared - bye *** glibc detected *** /usr/sbin/apache2: corrupted double-linked list: 0x00007fda105bae40 *** ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7fda0e8b1b96] /lib/x86_64-linux-gnu/libc.so.6(+0x7eec8)[0x7fda0e8b1ec8] /lib/x86_64-linux-gnu/libc.so.6(+0x80436)[0x7fda0e8b3436] /lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x75)[0x7fda0e8b5fb5] /lib/x86_64-linux-gnu/libc.so.6(+0xbb50b)[0x7fda0e8ee50b] /usr/lib/libapr-1.so.0(apr_dir_open+0x29)[0x7fda0ee24e76] /usr/sbin/apache2(ap_process_resource_config+0xed)[0x7fda0f71030d] /usr/sbin/apache2(+0x35aaa)[0x7fda0f702aaa] /usr/sbin/apache2(+0x40741)[0x7fda0f70d741] /usr/sbin/apache2(+0x42304)[0x7fda0f70f304] /usr/sbin/apache2(ap_build_config+0x9c)[0x7fda0f70f8cc] /usr/sbin/apache2(+0x42d01)[0x7fda0f70fd01] /usr/sbin/apache2(ap_process_resource_config+0x209)[0x7fda0f710429] /usr/sbin/apache2(ap_read_config+0x2df)[0x7fda0f71104f] /usr/sbin/apache2(main+0xbb6)[0x7fda0f6f93d6] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7fda0e85476d] /usr/sbin/apache2(+0x2c559)[0x7fda0f6f9559] ======= Memory map: ========" In the beginning you can see nice tries from bots trying to grab vulnerabilities: mofo bots tirelessly try to seek for admin consoles. Then at 07:41:47 restart requested, and apache segfaults. Crash dump is generated also. One question: what else can start regularly weekly which is not anacron? Any other advice is welcome, I am clueless at this moment. Thanks, Csaba -- -- You received this message because you are subscribed to the Google Groups "NLUG" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nlug-talk?hl=en --- You received this message because you are subscribed to the Google Groups "NLUG" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
