Try restarting apache yourself and check the logs. It is possible something is cycling apache (maybe for log rotation) and that FastCGI is preventing it from closing down cleanly. If you restart apache yourself and get the same error, it should lead you down the right path.
I will second stripping out everything you don't actively use. Not just for security purposes but for simplicity in management. Andy On Sun, Mar 10, 2013 at 9:18 PM, Toth, Csaba <[email protected]>wrote: > Hey Linux Gurus, > > I wonder about your advices how I can solve this. > Our lab's web server at Vandy crashes every Sunday at 7:41am exactly. > I cannot track down the culprit. It's an Ubuntu Server Edition 12.04.1 LTS > Precise. > Apache has php5 and even Drupal. We don't use Drupal and very little php. > > What I suspect that it's some scheduled task doing the harm. > The Anachron weekly starts at 7:30am. After that it start the cron.weekly > jobs with 10 minutes separation. Cron.weekly: > "0anacron > apt-xapian-index > debsums > man-db > sysklogd" > > 7:41 crash means that it can be the second thing it starts. I don't know > if 0anacron counts or not. Debsums: this is a package with which you can > identify damaged package. (The server had HDD hardware failure during the > summer. That was degrading for a while, and we had logical file system > issues too. I was down at the Medical Center DataCenter and I had to issue > more fscks to recover the filesystem fully. I'm even happy that I could > save the RAID volume, since it fell apart for sometime too.) > I went ahead and started all scripts in cron.weekly manually from a root > command prompt, but nothing caused a crash. More info about the crash: I > cannot see anything in the Apache access logs for 7:41am, the last log is > for 7:37am. > > Error log: > > "[Sun Mar 10 01:29:09 2013] [error] [client 108.222.66.23] File does not > exist: /var/www/www.hiplab.org/favicon.ico > [Sun Mar 10 01:50:02 2013] [error] [client 66.249.76.209] File does not > exist: /var/www/www.hiplab.org/people/malin/BMIF380 > [Sun Mar 10 03:36:04 2013] [error] [client 41.196.103.228] File does not > exist: /var/www/www.hiplab.org/favicon.ico > [Sun Mar 10 03:36:08 2013] [error] [client 41.196.103.228] File does not > exist: /var/www/www.hiplab.org/favicon.ico > [Sun Mar 10 03:36:40 2013] [error] [client 41.196.103.228] File does not > exist: /var/www/www.hiplab.org/favicon.ico > [Sun Mar 10 03:52:46 2013] [error] [client 199.30.20.73] File does not > exist: /var/www/www.hiplab.org/people/weiyixia/icon.jpg > [Sun Mar 10 04:46:58 2013] [error] [client 78.162.114.93] File does not > exist: /var/www/www.hiplab.org/favicon.ico > [Sun Mar 10 06:52:30 2013] [error] [client 69.163.180.32] File does not > exist: /var/www/www.hiplab.org/wp-admin > [Sun Mar 10 07:05:50 2013] [error] [client 69.163.187.202] File does not > exist: /var/www/www.hiplab.org/wordpress > [Sun Mar 10 07:26:48 2013] [error] [client 69.163.148.155] File does not > exist: /var/www/www.hiplab.org/wordpress > [Sun Mar 10 07:41:47 2013] [notice] Graceful restart requested, doing > restart > [Sun Mar 10 07:41:47 2013] [alert] (4)Interrupted system call: FastCGI: > read() from pipe failed (0) > [Sun Mar 10 07:41:47 2013] [alert] (4)Interrupted system call: FastCGI: > the PM is shutting down, Apache seems to have disappeared - bye > *** glibc detected *** /usr/sbin/apache2: corrupted double-linked list: > 0x00007fda105bae40 *** > ======= Backtrace: ========= > /lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7fda0e8b1b96] > /lib/x86_64-linux-gnu/libc.so.6(+0x7eec8)[0x7fda0e8b1ec8] > /lib/x86_64-linux-gnu/libc.so.6(+0x80436)[0x7fda0e8b3436] > /lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x75)[0x7fda0e8b5fb5] > /lib/x86_64-linux-gnu/libc.so.6(+0xbb50b)[0x7fda0e8ee50b] > /usr/lib/libapr-1.so.0(apr_dir_open+0x29)[0x7fda0ee24e76] > /usr/sbin/apache2(ap_process_resource_config+0xed)[0x7fda0f71030d] > /usr/sbin/apache2(+0x35aaa)[0x7fda0f702aaa] > /usr/sbin/apache2(+0x40741)[0x7fda0f70d741] > /usr/sbin/apache2(+0x42304)[0x7fda0f70f304] > /usr/sbin/apache2(ap_build_config+0x9c)[0x7fda0f70f8cc] > /usr/sbin/apache2(+0x42d01)[0x7fda0f70fd01] > /usr/sbin/apache2(ap_process_resource_config+0x209)[0x7fda0f710429] > /usr/sbin/apache2(ap_read_config+0x2df)[0x7fda0f71104f] > /usr/sbin/apache2(main+0xbb6)[0x7fda0f6f93d6] > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7fda0e85476d] > /usr/sbin/apache2(+0x2c559)[0x7fda0f6f9559] > ======= Memory map: ========" > > In the beginning you can see nice tries from bots trying to grab > vulnerabilities: mofo bots tirelessly try to seek for admin consoles. Then > at 07:41:47 restart requested, and apache segfaults. Crash dump is > generated also. > > One question: what else can start regularly weekly which is not anacron? > Any other advice is welcome, I am clueless at this moment. > > Thanks, > Csaba > > -- > -- > You received this message because you are subscribed to the Google Groups > "NLUG" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nlug-talk?hl=en > > --- > You received this message because you are subscribed to the Google Groups > "NLUG" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- -- You received this message because you are subscribed to the Google Groups "NLUG" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nlug-talk?hl=en --- You received this message because you are subscribed to the Google Groups "NLUG" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
