Here's a cure for the problem: look at the /etc/logrotate.d/apache2 file. It has a postrotate section, which does an /etc/init.d/apache2 reload. This reload crashes, I tried it manually too, and it causes the same crash I see on Sundays. In the logs we can see also, that something starts to restart apache: "Graceful restart requested, doing restart" before the crash. My cure is to simply remove that postrotate section and the apache2 reload. The whole issue was associated with the logrotate, this runs weekly. Apache2 restart works well, without a crash though. I tried to remove unnecessary stuff from apache config, because the real cause is why the apache crashes for a reload.
I can confirm my findings next Sunday morning. I have to say thanks for the best sysadmin I know, his everyday job is an MRI radiologist medical doctor: my twin brother Ati. I explained the problem and he knew the possible cause right away. Csaba ________________________________ From: [email protected] [[email protected]] on behalf of Michael Chaney [[email protected]] Sent: Tuesday, March 12, 2013 9:27 PM To: [email protected] Subject: Re: [nlug] Apache crashes every Sunday like a clockwork Here's a weird data point for you. I have an ubuntu server, probably version 8 or 10, that has a few rails sites running under lighttpd. It used to crash every two or three weeks at the same time. Guess what time it crashed. I never did track it down, I'd just restart it and go to church. Something with ubuntu. Michael On Sun, Mar 10, 2013 at 8:18 PM, Toth, Csaba <[email protected]<mailto:[email protected]>> wrote: Hey Linux Gurus, I wonder about your advices how I can solve this. Our lab's web server at Vandy crashes every Sunday at 7:41am exactly. I cannot track down the culprit. It's an Ubuntu Server Edition 12.04.1 LTS Precise. Apache has php5 and even Drupal. We don't use Drupal and very little php. What I suspect that it's some scheduled task doing the harm. The Anachron weekly starts at 7:30am. After that it start the cron.weekly jobs with 10 minutes separation. Cron.weekly: "0anacron apt-xapian-index debsums man-db sysklogd" 7:41 crash means that it can be the second thing it starts. I don't know if 0anacron counts or not. Debsums: this is a package with which you can identify damaged package. (The server had HDD hardware failure during the summer. That was degrading for a while, and we had logical file system issues too. I was down at the Medical Center DataCenter and I had to issue more fscks to recover the filesystem fully. I'm even happy that I could save the RAID volume, since it fell apart for sometime too.) I went ahead and started all scripts in cron.weekly manually from a root command prompt, but nothing caused a crash. More info about the crash: I cannot see anything in the Apache access logs for 7:41am, the last log is for 7:37am. Error log: "[Sun Mar 10 01:29:09 2013] [error] [client 108.222.66.23] File does not exist: /var/www/www.hiplab.org/favicon.ico<http://www.hiplab.org/favicon.ico> [Sun Mar 10 01:50:02 2013] [error] [client 66.249.76.209<tel:66.249.76.209>] File does not exist: /var/www/www.hiplab.org/people/malin/BMIF380<http://www.hiplab.org/people/malin/BMIF380> [Sun Mar 10 03:36:04 2013] [error] [client 41.196.103.228] File does not exist: /var/www/www.hiplab.org/favicon.ico<http://www.hiplab.org/favicon.ico> [Sun Mar 10 03:36:08 2013] [error] [client 41.196.103.228] File does not exist: /var/www/www.hiplab.org/favicon.ico<http://www.hiplab.org/favicon.ico> [Sun Mar 10 03:36:40 2013] [error] [client 41.196.103.228] File does not exist: /var/www/www.hiplab.org/favicon.ico<http://www.hiplab.org/favicon.ico> [Sun Mar 10 03:52:46 2013] [error] [client 199.30.20.73] File does not exist: /var/www/www.hiplab.org/people/weiyixia/icon.jpg<http://www.hiplab.org/people/weiyixia/icon.jpg> [Sun Mar 10 04:46:58 2013] [error] [client 78.162.114.93<tel:78.162.114.93>] File does not exist: /var/www/www.hiplab.org/favicon.ico<http://www.hiplab.org/favicon.ico> [Sun Mar 10 06:52:30 2013] [error] [client 69.163.180.32] File does not exist: /var/www/www.hiplab.org/wp-admin<http://www.hiplab.org/wp-admin> [Sun Mar 10 07:05:50 2013] [error] [client 69.163.187.202] File does not exist: /var/www/www.hiplab.org/wordpress<http://www.hiplab.org/wordpress> [Sun Mar 10 07:26:48 2013] [error] [client 69.163.148.155] File does not exist: /var/www/www.hiplab.org/wordpress<http://www.hiplab.org/wordpress> [Sun Mar 10 07:41:47 2013] [notice] Graceful restart requested, doing restart [Sun Mar 10 07:41:47 2013] [alert] (4)Interrupted system call: FastCGI: read() from pipe failed (0) [Sun Mar 10 07:41:47 2013] [alert] (4)Interrupted system call: FastCGI: the PM is shutting down, Apache seems to have disappeared - bye *** glibc detected *** /usr/sbin/apache2: corrupted double-linked list: 0x00007fda105bae40 *** ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7fda0e8b1b96] /lib/x86_64-linux-gnu/libc.so.6(+0x7eec8)[0x7fda0e8b1ec8] /lib/x86_64-linux-gnu/libc.so.6(+0x80436)[0x7fda0e8b3436] /lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x75)[0x7fda0e8b5fb5] /lib/x86_64-linux-gnu/libc.so.6(+0xbb50b)[0x7fda0e8ee50b] /usr/lib/libapr-1.so.0(apr_dir_open+0x29)[0x7fda0ee24e76] /usr/sbin/apache2(ap_process_resource_config+0xed)[0x7fda0f71030d] /usr/sbin/apache2(+0x35aaa)[0x7fda0f702aaa] /usr/sbin/apache2(+0x40741)[0x7fda0f70d741] /usr/sbin/apache2(+0x42304)[0x7fda0f70f304] /usr/sbin/apache2(ap_build_config+0x9c)[0x7fda0f70f8cc] /usr/sbin/apache2(+0x42d01)[0x7fda0f70fd01] /usr/sbin/apache2(ap_process_resource_config+0x209)[0x7fda0f710429] /usr/sbin/apache2(ap_read_config+0x2df)[0x7fda0f71104f] /usr/sbin/apache2(main+0xbb6)[0x7fda0f6f93d6] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7fda0e85476d] /usr/sbin/apache2(+0x2c559)[0x7fda0f6f9559] ======= Memory map: ========" In the beginning you can see nice tries from bots trying to grab vulnerabilities: mofo bots tirelessly try to seek for admin consoles. Then at 07:41:47 restart requested, and apache segfaults. Crash dump is generated also. One question: what else can start regularly weekly which is not anacron? Any other advice is welcome, I am clueless at this moment. Thanks, Csaba -- -- You received this message because you are subscribed to the Google Groups "NLUG" group. To post to this group, send email to [email protected]<mailto:[email protected]> To unsubscribe from this group, send email to [email protected]<mailto:nlug-talk%[email protected]> For more options, visit this group at http://groups.google.com/group/nlug-talk?hl=en --- You received this message because you are subscribed to the Google Groups "NLUG" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:nlug-talk%[email protected]>. For more options, visit https://groups.google.com/groups/opt_out. -- Michael Darrin Chaney, Sr. [email protected]<mailto:[email protected]> http://www.michaelchaney.com/ -- -- You received this message because you are subscribed to the Google Groups "NLUG" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nlug-talk?hl=en --- You received this message because you are subscribed to the Google Groups "NLUG" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out. -- -- You received this message because you are subscribed to the Google Groups "NLUG" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nlug-talk?hl=en --- You received this message because you are subscribed to the Google Groups "NLUG" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
