On Sun, Mar 10, 2013 at 8:18 PM, Toth, Csaba <[email protected]> wrote: > Hey Linux Gurus, > > I wonder about your advices how I can solve this. > Our lab's web server at Vandy crashes every Sunday at 7:41am exactly. > I cannot track down the culprit. It's an Ubuntu Server Edition 12.04.1 LTS > Precise. > Apache has php5 and even Drupal. We don't use Drupal and very little php. > > What I suspect that it's some scheduled task doing the harm. > The Anachron weekly starts at 7:30am. After that it start the cron.weekly > jobs with 10 minutes separation. Cron.weekly: > "0anacron > apt-xapian-index
Is it possible that apt-xapian-index is causing a cpu spike and somehow causing apache to have that behavior? Do you have any logs on xapian? http://forums.linuxmint.com/viewtopic.php?f=90&t=82985 Andrew McElroy > debsums > man-db > sysklogd" > > 7:41 crash means that it can be the second thing it starts. I don't know if > 0anacron counts or not. Debsums: this is a package with which you can > identify damaged package. (The server had HDD hardware failure during the > summer. That was degrading for a while, and we had logical file system > issues too. I was down at the Medical Center DataCenter and I had to issue > more fscks to recover the filesystem fully. I'm even happy that I could save > the RAID volume, since it fell apart for sometime too.) > I went ahead and started all scripts in cron.weekly manually from a root > command prompt, but nothing caused a crash. More info about the crash: I > cannot see anything in the Apache access logs for 7:41am, the last log is > for 7:37am. > > Error log: > > "[Sun Mar 10 01:29:09 2013] [error] [client 108.222.66.23] File does not > exist: /var/www/www.hiplab.org/favicon.ico > [Sun Mar 10 01:50:02 2013] [error] [client 66.249.76.209] File does not > exist: /var/www/www.hiplab.org/people/malin/BMIF380 > [Sun Mar 10 03:36:04 2013] [error] [client 41.196.103.228] File does not > exist: /var/www/www.hiplab.org/favicon.ico > [Sun Mar 10 03:36:08 2013] [error] [client 41.196.103.228] File does not > exist: /var/www/www.hiplab.org/favicon.ico > [Sun Mar 10 03:36:40 2013] [error] [client 41.196.103.228] File does not > exist: /var/www/www.hiplab.org/favicon.ico > [Sun Mar 10 03:52:46 2013] [error] [client 199.30.20.73] File does not > exist: /var/www/www.hiplab.org/people/weiyixia/icon.jpg > [Sun Mar 10 04:46:58 2013] [error] [client 78.162.114.93] File does not > exist: /var/www/www.hiplab.org/favicon.ico > [Sun Mar 10 06:52:30 2013] [error] [client 69.163.180.32] File does not > exist: /var/www/www.hiplab.org/wp-admin > [Sun Mar 10 07:05:50 2013] [error] [client 69.163.187.202] File does not > exist: /var/www/www.hiplab.org/wordpress > [Sun Mar 10 07:26:48 2013] [error] [client 69.163.148.155] File does not > exist: /var/www/www.hiplab.org/wordpress > [Sun Mar 10 07:41:47 2013] [notice] Graceful restart requested, doing > restart > [Sun Mar 10 07:41:47 2013] [alert] (4)Interrupted system call: FastCGI: > read() from pipe failed (0) > [Sun Mar 10 07:41:47 2013] [alert] (4)Interrupted system call: FastCGI: the > PM is shutting down, Apache seems to have disappeared - bye > *** glibc detected *** /usr/sbin/apache2: corrupted double-linked list: > 0x00007fda105bae40 *** > ======= Backtrace: ========= > /lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7fda0e8b1b96] > /lib/x86_64-linux-gnu/libc.so.6(+0x7eec8)[0x7fda0e8b1ec8] > /lib/x86_64-linux-gnu/libc.so.6(+0x80436)[0x7fda0e8b3436] > /lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x75)[0x7fda0e8b5fb5] > /lib/x86_64-linux-gnu/libc.so.6(+0xbb50b)[0x7fda0e8ee50b] > /usr/lib/libapr-1.so.0(apr_dir_open+0x29)[0x7fda0ee24e76] > /usr/sbin/apache2(ap_process_resource_config+0xed)[0x7fda0f71030d] > /usr/sbin/apache2(+0x35aaa)[0x7fda0f702aaa] > /usr/sbin/apache2(+0x40741)[0x7fda0f70d741] > /usr/sbin/apache2(+0x42304)[0x7fda0f70f304] > /usr/sbin/apache2(ap_build_config+0x9c)[0x7fda0f70f8cc] > /usr/sbin/apache2(+0x42d01)[0x7fda0f70fd01] > /usr/sbin/apache2(ap_process_resource_config+0x209)[0x7fda0f710429] > /usr/sbin/apache2(ap_read_config+0x2df)[0x7fda0f71104f] > /usr/sbin/apache2(main+0xbb6)[0x7fda0f6f93d6] > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7fda0e85476d] > /usr/sbin/apache2(+0x2c559)[0x7fda0f6f9559] > ======= Memory map: ========" > > In the beginning you can see nice tries from bots trying to grab > vulnerabilities: mofo bots tirelessly try to seek for admin consoles. Then > at 07:41:47 restart requested, and apache segfaults. Crash dump is generated > also. > > One question: what else can start regularly weekly which is not anacron? > Any other advice is welcome, I am clueless at this moment. > > Thanks, > Csaba > > -- > -- > You received this message because you are subscribed to the Google Groups > "NLUG" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nlug-talk?hl=en > > --- > You received this message because you are subscribed to the Google Groups > "NLUG" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > > -- -- You received this message because you are subscribed to the Google Groups "NLUG" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nlug-talk?hl=en --- You received this message because you are subscribed to the Google Groups "NLUG" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
