Re: Server rebooted at 3 a.m. and 7 a.m. for the past few days
On Friday 15 August 2003 02:16, Magnus J wrote: Hello Running /usr/local/etc/cvsup/update.sh manually caused the machine to reboot. Unfortunately, /var/log/cvsup.log doesn't provide any information about why. Any recommendation on what I should use to get more messages? Have you tried recompiling your kernel with: options DDB makeotions DEBUG=-g in your kernel config file? BTW, what version of FreeBSD are you running? Best regards, Daniela ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: Server rebooted at 3 a.m. and 7 a.m. for the past few days
Hello I tried to start up my Distributed Folding client, which consumes quite a lot of CPU, and it didn't take long before the machine rebooted. I guess the CPU fan is the first thing I will look at when I get back home. This is a PIII 850 MHz Slot 1. I've never had any problems with them before, but maybe others have a different experience. Thanks again to everyone who has responded. Magnus --- Brent Wiese [EMAIL PROTECTED] skrev: There are several system utils that'll stress the CPU/disk in the ports section. I'd try some of those to see if you can cause a reboot. If so, it might help diagnose... If you have a bad cpu fan, it doesn't take much to crash the box. I've seen this a lot in older dual p2/p3 box style cpus. The fan on the cpu who's less than a finger-width from the other CPU siezes up. The box will run fine under no load, but as soon as you put any kind of load on the box, it dies. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Magnus J Sent: Thursday, August 14, 2003 7:16 PM To: Luke Kearney Cc: [EMAIL PROTECTED] Subject: Re: Server rebooted at 3 a.m. and 7 a.m. for the past few days Hello Running /usr/local/etc/cvsup/update.sh manually caused the machine to reboot. Unfortunately, /var/log/cvsup.log doesn't provide any information about why. Any recommendation on what I should use to get more messages? Thanks Magnus --- Luke Kearney [EMAIL PROTECTED] skrev: - Original Message - From: Magnus J [EMAIL PROTECTED] To: Brent Wiese [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Sent: Friday, August 15, 2003 10:44 AM Subject: RE: Server rebooted at 3 a.m. and 7 a.m. for the past few days Hello dmesg shows no panic, and nothing that consumes much CPU has been running since the first reboot. Around 3 a.m. the daily periodic runs (which is default) and around 7 a.m. cvsup runs. Thanks Magnus --- Brent Wiese [EMAIL PROTECTED] skrev: Do you have any scripts that run at those times? If you run something like a database update or something that can crank some CPU cycles, you could be overheating the box, causing a reboot. Could happen all of a sudden if a fan decided to quit... Dmesg show any panics? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Magnus J Sent: Thursday, August 14, 2003 5:22 PM To: Steve Hovey Cc: [EMAIL PROTECTED] Subject: Re: Server rebooted at 3 a.m. and 7 a.m. for the past few days Hello Thanks for replying. /etc/crontab looks OK. This is how 'last' looks like (user1 is myself) user1 ttyp0zzz.12.28.40 Thu Aug 14 12:43 - 13:30 (00:46) user1 ttyp1zzz.12.28.40 Thu Aug 14 12:20 - 13:30 (01:09) user1 ttyp0zzz.12.28.40 Thu Aug 14 12:08 - 12:21 (00:12) user1 ttyp0zzz.12.27.12 Thu Aug 14 10:06 - 11:22 (01:15) user1 ttyp1zzz.12.28.52 Thu Aug 14 08:06 - 08:07 (00:00) user1 ttyp0zzz.12.28.52 Thu Aug 14 07:10 - 08:07 (00:56) reboot ~ Thu Aug 14 07:10 reboot ~ Thu Aug 14 03:09 reboot ~ Wed Aug 13 07:13 reboot ~ Wed Aug 13 03:09 reboot ~ Tue Aug 12 07:12 reboot ~ Tue Aug 12 03:09 reboot ~ Mon Aug 11 07:11 reboot ~ Mon Aug 11 03:09 reboot ~ Sun Aug 10 07:10 reboot ~ Sun Aug 10 03:08 reboot ~ Sat Aug 9 07:10 reboot ~ Sat Aug 9 04:22 reboot ~ Sat Aug 9 03:08 reboot ~ Fri Aug 8 07:10 reboot ~ Thu Aug 7 22:21 user1 ttyp4zzz.12.28.14 Mon Aug 4 22:39 - 22:40 (00:00) wtmp begins Mon Aug 4 22:39:55 CEST 2003 bash-2.05b# date Fri Aug 15 02:06:22 CEST 2003 bash-2.05b# Should I worry about these messages? Jul 16 14:06:47 magnus1 sshd[22292]: scanned from zzz.7.104.10 with SSH-1.0-SSH_ Version_Mapper. Don't panic. Jul 16 14:06:47 magnus1 sshd[22291]: Did not receive
Server rebooted at 3 a.m. and 7 a.m. for the past few days
Hello everyone I'm not sure if I should have posted this to freebsd-security, but I start here. I'm out traveling, and finally got a chance to login to my server back home through SSH, which is running 4.8 and is protected by an IPFILTER firewall. Looking at /var/log/messages , the server has been mysteriously rebooted around 3 a.m. and 7 a.m. CET every day for the past few days. I have never seen this before. It doesn't look like hardware problem because it's not random and there are no messages about filesystems not being unmounted cleanly. Any ideas where I should start looking to see what's going on? Obviously I will try to monitor what's happening next time around 3 a.m. and 7 a.m., which processes are running, etc., but is there something special I should look out for? Unfortunately, I have not installed Tripwire. Best regards Magnus (not a member of this list) Yahoo! Mail - Gratis: 6 MB lagringsutrymme, spamfilter och virusscan. Se mer på http://se.mail.yahoo.com ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Server rebooted at 3 a.m. and 7 a.m. for the past few days
Hello Thanks for replying. /etc/crontab looks OK. This is how 'last' looks like (user1 is myself) user1 ttyp0zzz.12.28.40 Thu Aug 14 12:43 - 13:30 (00:46) user1 ttyp1zzz.12.28.40 Thu Aug 14 12:20 - 13:30 (01:09) user1 ttyp0zzz.12.28.40 Thu Aug 14 12:08 - 12:21 (00:12) user1 ttyp0zzz.12.27.12 Thu Aug 14 10:06 - 11:22 (01:15) user1 ttyp1zzz.12.28.52 Thu Aug 14 08:06 - 08:07 (00:00) user1 ttyp0zzz.12.28.52 Thu Aug 14 07:10 - 08:07 (00:56) reboot ~ Thu Aug 14 07:10 reboot ~ Thu Aug 14 03:09 reboot ~ Wed Aug 13 07:13 reboot ~ Wed Aug 13 03:09 reboot ~ Tue Aug 12 07:12 reboot ~ Tue Aug 12 03:09 reboot ~ Mon Aug 11 07:11 reboot ~ Mon Aug 11 03:09 reboot ~ Sun Aug 10 07:10 reboot ~ Sun Aug 10 03:08 reboot ~ Sat Aug 9 07:10 reboot ~ Sat Aug 9 04:22 reboot ~ Sat Aug 9 03:08 reboot ~ Fri Aug 8 07:10 reboot ~ Thu Aug 7 22:21 user1 ttyp4zzz.12.28.14 Mon Aug 4 22:39 - 22:40 (00:00) wtmp begins Mon Aug 4 22:39:55 CEST 2003 bash-2.05b# date Fri Aug 15 02:06:22 CEST 2003 bash-2.05b# Should I worry about these messages? Jul 16 14:06:47 magnus1 sshd[22292]: scanned from zzz.7.104.10 with SSH-1.0-SSH_ Version_Mapper. Don't panic. Jul 16 14:06:47 magnus1 sshd[22291]: Did not receive identification string from zzz.7.104.10 Jul 27 19:58:36 magnus1 sshd[1811]: scanned from zzz.18.53.102 with SSH-1.0-SSH_Ve Jul 27 19:58:36 magnus1 sshd[1811]: scanned from zzz.18.53.102 with SSH-1.0-SSH_Ve rsion_Mapper. Don't panic. Jul 27 19:58:36 magnus1 sshd[1810]: Did not receive identification string from zzz.18.53.102 Jul 28 07:00:07 magnus1 sshd[2568]: Did not receive identification string from zzz.155.91.132 Jul 29 05:59:55 magnus1 sshd[3798]: Did not receive identification string from zzz.235.37.77 Jul 30 10:53:55 magnus1 sshd[5285]: Did not receive identification string from zzz.111.110.6 Jul 30 10:56:51 magnus1 sshd[5289]: Did not receive identification string from zzz.111.110.6 Jul 30 12:51:46 magnus1 sshd[5365]: Did not receive identification string from zzz.212.236.18 Jul 31 02:57:59 magnus1 sshd[5935]: Did not receive identification string from zzz.30.187.2 Aug 4 08:15:11 magnus1 sshd[14242]: Did not receive identification string from zzz.246.43.167 Previously, I have had easily two months of uptime on this server. Regards Magnus --- Steve Hovey [EMAIL PROTECTED] skrev: I would start with your cron jobs On Thu, 14 Aug 2003, [iso-8859-1] Magnus J wrote: Hello everyone I'm not sure if I should have posted this to freebsd-security, but I start here. I'm out traveling, and finally got a chance to login to my server back home through SSH, which is running 4.8 and is protected by an IPFILTER firewall. Looking at /var/log/messages , the server has been mysteriously rebooted around 3 a.m. and 7 a.m. CET every day for the past few days. I have never seen this before. It doesn't look like hardware problem because it's not random and there are no messages about filesystems not being unmounted cleanly. Any ideas where I should start looking to see what's going on? Obviously I will try to monitor what's happening next time around 3 a.m. and 7 a.m., which processes are running, etc., but is there something special I should look out for? Unfortunately, I have not installed Tripwire. Best regards Magnus (not a member of this list) Yahoo! Mail - Gratis: 6 MB lagringsutrymme, spamfilter och virusscan. Se mer på http://se.mail.yahoo.com ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] Yahoo! Mail - Gratis: 6 MB lagringsutrymme, spamfilter och virusscan. Se mer på http://se.mail.yahoo.com ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
RE: Server rebooted at 3 a.m. and 7 a.m. for the past few days
Do you have any scripts that run at those times? If you run something like a database update or something that can crank some CPU cycles, you could be overheating the box, causing a reboot. Could happen all of a sudden if a fan decided to quit... Dmesg show any panics? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Magnus J Sent: Thursday, August 14, 2003 5:22 PM To: Steve Hovey Cc: [EMAIL PROTECTED] Subject: Re: Server rebooted at 3 a.m. and 7 a.m. for the past few days Hello Thanks for replying. /etc/crontab looks OK. This is how 'last' looks like (user1 is myself) user1 ttyp0zzz.12.28.40 Thu Aug 14 12:43 - 13:30 (00:46) user1 ttyp1zzz.12.28.40 Thu Aug 14 12:20 - 13:30 (01:09) user1 ttyp0zzz.12.28.40 Thu Aug 14 12:08 - 12:21 (00:12) user1 ttyp0zzz.12.27.12 Thu Aug 14 10:06 - 11:22 (01:15) user1 ttyp1zzz.12.28.52 Thu Aug 14 08:06 - 08:07 (00:00) user1 ttyp0zzz.12.28.52 Thu Aug 14 07:10 - 08:07 (00:56) reboot ~ Thu Aug 14 07:10 reboot ~ Thu Aug 14 03:09 reboot ~ Wed Aug 13 07:13 reboot ~ Wed Aug 13 03:09 reboot ~ Tue Aug 12 07:12 reboot ~ Tue Aug 12 03:09 reboot ~ Mon Aug 11 07:11 reboot ~ Mon Aug 11 03:09 reboot ~ Sun Aug 10 07:10 reboot ~ Sun Aug 10 03:08 reboot ~ Sat Aug 9 07:10 reboot ~ Sat Aug 9 04:22 reboot ~ Sat Aug 9 03:08 reboot ~ Fri Aug 8 07:10 reboot ~ Thu Aug 7 22:21 user1 ttyp4zzz.12.28.14 Mon Aug 4 22:39 - 22:40 (00:00) wtmp begins Mon Aug 4 22:39:55 CEST 2003 bash-2.05b# date Fri Aug 15 02:06:22 CEST 2003 bash-2.05b# Should I worry about these messages? Jul 16 14:06:47 magnus1 sshd[22292]: scanned from zzz.7.104.10 with SSH-1.0-SSH_ Version_Mapper. Don't panic. Jul 16 14:06:47 magnus1 sshd[22291]: Did not receive identification string from zzz.7.104.10 Jul 27 19:58:36 magnus1 sshd[1811]: scanned from zzz.18.53.102 with SSH-1.0-SSH_Ve Jul 27 19:58:36 magnus1 sshd[1811]: scanned from zzz.18.53.102 with SSH-1.0-SSH_Ve rsion_Mapper. Don't panic. Jul 27 19:58:36 magnus1 sshd[1810]: Did not receive identification string from zzz.18.53.102 Jul 28 07:00:07 magnus1 sshd[2568]: Did not receive identification string from zzz.155.91.132 Jul 29 05:59:55 magnus1 sshd[3798]: Did not receive identification string from zzz.235.37.77 Jul 30 10:53:55 magnus1 sshd[5285]: Did not receive identification string from zzz.111.110.6 Jul 30 10:56:51 magnus1 sshd[5289]: Did not receive identification string from zzz.111.110.6 Jul 30 12:51:46 magnus1 sshd[5365]: Did not receive identification string from zzz.212.236.18 Jul 31 02:57:59 magnus1 sshd[5935]: Did not receive identification string from zzz.30.187.2 Aug 4 08:15:11 magnus1 sshd[14242]: Did not receive identification string from zzz.246.43.167 Previously, I have had easily two months of uptime on this server. Regards Magnus --- Steve Hovey [EMAIL PROTECTED] skrev: I would start with your cron jobs On Thu, 14 Aug 2003, [iso-8859-1] Magnus J wrote: Hello everyone I'm not sure if I should have posted this to freebsd-security, but I start here. I'm out traveling, and finally got a chance to login to my server back home through SSH, which is running 4.8 and is protected by an IPFILTER firewall. Looking at /var/log/messages , the server has been mysteriously rebooted around 3 a.m. and 7 a.m. CET every day for the past few days. I have never seen this before. It doesn't look like hardware problem because it's not random and there are no messages about filesystems not being unmounted cleanly. Any ideas where I should start looking to see what's going on? Obviously I will try to monitor what's happening next time around 3 a.m. and 7 a.m., which processes are running, etc., but is there something special I should look out for? Unfortunately, I have not installed Tripwire. Best regards Magnus (not a member of this list) Yahoo! Mail - Gratis: 6 MB lagringsutrymme, spamfilter och virusscan. Se mer på http://se.mail.yahoo.com ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL
Re: Server rebooted at 3 a.m. and 7 a.m. for the past few days
Hello sockstat -4 didn't show anything unusual: sshd, cvsupd, java, portmap and ntpd. It seems as if the reboots are happening during the daily periodic and when cvsup runs around 7 a.m. in the morning. I just monitored before the 3 a.m. reboot, and there was no reboot message printed when I lost connection. I could see that find as part of the security check had started. After looking again in the boot-up messages - / was not properly dismounted, but no message(s) for the other partitions. find and cvsup both do disk access. Could it be that one of the disks are bad causing the system to crash, or am I way off there? Thanks Magnus --- Luke Kearney [EMAIL PROTECTED] skrev: hello, I am sorry if this seems a bit obvious and silly but try sockstat -4 to see if there are any *new* services running or ports listening that you would not normally have running. If you have been cracked then that is a good place to look. the regular reboots are a concern. My boxes usually only get rebooted once a year so you should be able to expect well in excess of 2 to 3 mths without issue HTH - Original Message - From: Magnus J [EMAIL PROTECTED] To: Steve Hovey [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Sent: Friday, August 15, 2003 9:22 AM Subject: Re: Server rebooted at 3 a.m. and 7 a.m. for the past few days Hello Thanks for replying. /etc/crontab looks OK. This is how 'last' looks like (user1 is myself) user1 ttyp0zzz.12.28.40 Thu Aug 14 12:43 - 13:30 (00:46) user1 ttyp1zzz.12.28.40 Thu Aug 14 12:20 - 13:30 (01:09) user1 ttyp0zzz.12.28.40 Thu Aug 14 12:08 - 12:21 (00:12) user1 ttyp0zzz.12.27.12 Thu Aug 14 10:06 - 11:22 (01:15) user1 ttyp1zzz.12.28.52 Thu Aug 14 08:06 - 08:07 (00:00) user1 ttyp0zzz.12.28.52 Thu Aug 14 07:10 - 08:07 (00:56) reboot ~ Thu Aug 14 07:10 reboot ~ Thu Aug 14 03:09 reboot ~ Wed Aug 13 07:13 reboot ~ Wed Aug 13 03:09 reboot ~ Tue Aug 12 07:12 reboot ~ Tue Aug 12 03:09 reboot ~ Mon Aug 11 07:11 reboot ~ Mon Aug 11 03:09 reboot ~ Sun Aug 10 07:10 reboot ~ Sun Aug 10 03:08 reboot ~ Sat Aug 9 07:10 reboot ~ Sat Aug 9 04:22 reboot ~ Sat Aug 9 03:08 reboot ~ Fri Aug 8 07:10 reboot ~ Thu Aug 7 22:21 user1 ttyp4zzz.12.28.14 Mon Aug 4 22:39 - 22:40 (00:00) wtmp begins Mon Aug 4 22:39:55 CEST 2003 bash-2.05b# date Fri Aug 15 02:06:22 CEST 2003 bash-2.05b# Should I worry about these messages? Jul 16 14:06:47 magnus1 sshd[22292]: scanned from zzz.7.104.10 with SSH-1.0-SSH_ Version_Mapper. Don't panic. Jul 16 14:06:47 magnus1 sshd[22291]: Did not receive identification string from zzz.7.104.10 Jul 27 19:58:36 magnus1 sshd[1811]: scanned from zzz.18.53.102 with SSH-1.0-SSH_Ve Jul 27 19:58:36 magnus1 sshd[1811]: scanned from zzz.18.53.102 with SSH-1.0-SSH_Ve rsion_Mapper. Don't panic. Jul 27 19:58:36 magnus1 sshd[1810]: Did not receive identification string from zzz.18.53.102 Jul 28 07:00:07 magnus1 sshd[2568]: Did not receive identification string from zzz.155.91.132 Jul 29 05:59:55 magnus1 sshd[3798]: Did not receive identification string from zzz.235.37.77 Jul 30 10:53:55 magnus1 sshd[5285]: Did not receive identification string from zzz.111.110.6 Jul 30 10:56:51 magnus1 sshd[5289]: Did not receive identification string from zzz.111.110.6 Jul 30 12:51:46 magnus1 sshd[5365]: Did not receive identification string from zzz.212.236.18 Jul 31 02:57:59 magnus1 sshd[5935]: Did not receive identification string from zzz.30.187.2 Aug 4 08:15:11 magnus1 sshd[14242]: Did not receive identification string from zzz.246.43.167 Previously, I have had easily two months of uptime on this server. Regards Magnus --- Steve Hovey [EMAIL PROTECTED] skrev: I would start with your cron jobs On Thu, 14 Aug 2003, [iso-8859-1] Magnus J wrote: Hello everyone I'm not sure if I should have posted this to freebsd-security, but I start here. I'm out traveling, and finally got a chance to login to my server back home through SSH, which is running 4.8 and is protected by an IPFILTER firewall. Looking at /var/log/messages , the server has been mysteriously rebooted around 3 a.m. and 7 a.m. CET every day for the past
RE: Server rebooted at 3 a.m. and 7 a.m. for the past few days
Hello dmesg shows no panic, and nothing that consumes much CPU has been running since the first reboot. Around 3 a.m. the daily periodic runs (which is default) and around 7 a.m. cvsup runs. Thanks Magnus --- Brent Wiese [EMAIL PROTECTED] skrev: Do you have any scripts that run at those times? If you run something like a database update or something that can crank some CPU cycles, you could be overheating the box, causing a reboot. Could happen all of a sudden if a fan decided to quit... Dmesg show any panics? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Magnus J Sent: Thursday, August 14, 2003 5:22 PM To: Steve Hovey Cc: [EMAIL PROTECTED] Subject: Re: Server rebooted at 3 a.m. and 7 a.m. for the past few days Hello Thanks for replying. /etc/crontab looks OK. This is how 'last' looks like (user1 is myself) user1 ttyp0zzz.12.28.40 Thu Aug 14 12:43 - 13:30 (00:46) user1 ttyp1zzz.12.28.40 Thu Aug 14 12:20 - 13:30 (01:09) user1 ttyp0zzz.12.28.40 Thu Aug 14 12:08 - 12:21 (00:12) user1 ttyp0zzz.12.27.12 Thu Aug 14 10:06 - 11:22 (01:15) user1 ttyp1zzz.12.28.52 Thu Aug 14 08:06 - 08:07 (00:00) user1 ttyp0zzz.12.28.52 Thu Aug 14 07:10 - 08:07 (00:56) reboot ~ Thu Aug 14 07:10 reboot ~ Thu Aug 14 03:09 reboot ~ Wed Aug 13 07:13 reboot ~ Wed Aug 13 03:09 reboot ~ Tue Aug 12 07:12 reboot ~ Tue Aug 12 03:09 reboot ~ Mon Aug 11 07:11 reboot ~ Mon Aug 11 03:09 reboot ~ Sun Aug 10 07:10 reboot ~ Sun Aug 10 03:08 reboot ~ Sat Aug 9 07:10 reboot ~ Sat Aug 9 04:22 reboot ~ Sat Aug 9 03:08 reboot ~ Fri Aug 8 07:10 reboot ~ Thu Aug 7 22:21 user1 ttyp4zzz.12.28.14 Mon Aug 4 22:39 - 22:40 (00:00) wtmp begins Mon Aug 4 22:39:55 CEST 2003 bash-2.05b# date Fri Aug 15 02:06:22 CEST 2003 bash-2.05b# Should I worry about these messages? Jul 16 14:06:47 magnus1 sshd[22292]: scanned from zzz.7.104.10 with SSH-1.0-SSH_ Version_Mapper. Don't panic. Jul 16 14:06:47 magnus1 sshd[22291]: Did not receive identification string from zzz.7.104.10 Jul 27 19:58:36 magnus1 sshd[1811]: scanned from zzz.18.53.102 with SSH-1.0-SSH_Ve Jul 27 19:58:36 magnus1 sshd[1811]: scanned from zzz.18.53.102 with SSH-1.0-SSH_Ve rsion_Mapper. Don't panic. Jul 27 19:58:36 magnus1 sshd[1810]: Did not receive identification string from zzz.18.53.102 Jul 28 07:00:07 magnus1 sshd[2568]: Did not receive identification string from zzz.155.91.132 Jul 29 05:59:55 magnus1 sshd[3798]: Did not receive identification string from zzz.235.37.77 Jul 30 10:53:55 magnus1 sshd[5285]: Did not receive identification string from zzz.111.110.6 Jul 30 10:56:51 magnus1 sshd[5289]: Did not receive identification string from zzz.111.110.6 Jul 30 12:51:46 magnus1 sshd[5365]: Did not receive identification string from zzz.212.236.18 Jul 31 02:57:59 magnus1 sshd[5935]: Did not receive identification string from zzz.30.187.2 Aug 4 08:15:11 magnus1 sshd[14242]: Did not receive identification string from zzz.246.43.167 Previously, I have had easily two months of uptime on this server. Regards Magnus --- Steve Hovey [EMAIL PROTECTED] skrev: I would start with your cron jobs On Thu, 14 Aug 2003, [iso-8859-1] Magnus J wrote: Hello everyone I'm not sure if I should have posted this to freebsd-security, but I start here. I'm out traveling, and finally got a chance to login to my server back home through SSH, which is running 4.8 and is protected by an IPFILTER firewall. Looking at /var/log/messages , the server has been mysteriously rebooted around 3 a.m. and 7 a.m. CET every day for the past few days. I have never seen this before. It doesn't look like hardware problem because it's not random and there are no messages about filesystems not being unmounted cleanly. Any ideas where I should start looking to see what's going on? Obviously I will try to monitor what's happening next time around 3 a.m. and 7 a.m., which processes are running, etc., but is there something special I should look out for? Unfortunately, I have not installed
Re: Server rebooted at 3 a.m. and 7 a.m. for the past few days
- Original Message - (BFrom: "Magnus J" [EMAIL PROTECTED] (BTo: "Brent Wiese" [EMAIL PROTECTED] (BCc: [EMAIL PROTECTED] (BSent: Friday, August 15, 2003 10:44 AM (BSubject: RE: Server rebooted at 3 a.m. and 7 a.m. for the past few days (B (B (B Hello (B (B (B dmesg shows no panic, and nothing that consumes much CPU has (B been running since the first reboot. (B Around 3 a.m. the daily periodic runs (which is default) and (B around 7 a.m. cvsup runs. (B (B Thanks (B Magnus (B (B --- Brent Wiese [EMAIL PROTECTED] skrev: (B Do you have any scripts that run at those times? If you run (B something like a (B database update or something that can crank some CPU cycles, (B you could be (B overheating the box, causing a reboot. Could happen "all of a (B sudden" if a (B fan decided to quit... (B (B Dmesg show any panics? (B (B -Original Message- (B From: [EMAIL PROTECTED] (B [mailto:[EMAIL PROTECTED] On Behalf Of (B Magnus J (B Sent: Thursday, August 14, 2003 5:22 PM (B To: Steve Hovey (B Cc: [EMAIL PROTECTED] (B Subject: Re: Server rebooted at 3 a.m. and 7 a.m. for the (B past few days (B (B (B Hello (B (B (B Thanks for replying. /etc/crontab looks OK. (B (B This is how 'last' looks like (user1 is myself) (B (B user1 ttyp0zzz.12.28.40 Thu Aug 14 12:43 (B - (B 13:30 (00:46) (B user1 ttyp1zzz.12.28.40 Thu Aug 14 12:20 (B - (B 13:30 (01:09) (B user1 ttyp0zzz.12.28.40 Thu Aug 14 12:08 (B - (B 12:21 (00:12) (B user1 ttyp0zzz.12.27.12 Thu Aug 14 10:06 (B - (B 11:22 (01:15) (B user1 ttyp1zzz.12.28.52 Thu Aug 14 08:06 (B - (B 08:07 (00:00) (B user1 ttyp0zzz.12.28.52 Thu Aug 14 07:10 (B - (B 08:07 (00:56) (B reboot ~ Thu Aug 14 07:10 (B reboot ~ Thu Aug 14 03:09 (B reboot ~ Wed Aug 13 07:13 (B reboot ~ Wed Aug 13 03:09 (B reboot ~ Tue Aug 12 07:12 (B reboot ~ Tue Aug 12 03:09 (B reboot ~ Mon Aug 11 07:11 (B reboot ~ Mon Aug 11 03:09 (B reboot ~ Sun Aug 10 07:10 (B reboot ~ Sun Aug 10 03:08 (B reboot ~ Sat Aug 9 07:10 (B reboot ~ Sat Aug 9 04:22 (B reboot ~ Sat Aug 9 03:08 (B reboot ~ Fri Aug 8 07:10 (B reboot ~ Thu Aug 7 22:21 (B user1 ttyp4zzz.12.28.14 Mon Aug 4 22:39 (B - (B 22:40 (00:00) (B (B wtmp begins Mon Aug 4 22:39:55 CEST 2003 (B bash-2.05b# date (B Fri Aug 15 02:06:22 CEST 2003 (B bash-2.05b# (B (B Should I worry about these messages? (B (B Jul 16 14:06:47 magnus1 sshd[22292]: scanned from (B zzz.7.104.10 with SSH-1.0-SSH_ Version_Mapper. Don't panic. (B (B Jul 16 14:06:47 magnus1 sshd[22291]: Did not receive (B identification string from zzz.7.104.10 Jul 27 19:58:36 (B magnus1 sshd[1811]: scanned from zzz.18.53.102 with (B SSH-1.0-SSH_Ve Jul 27 19:58:36 magnus1 sshd[1811]: scanned (B from zzz.18.53.102 with SSH-1.0-SSH_Ve rsion_Mapper. Don't (B panic. Jul 27 19:58:36 magnus1 sshd[1810]: Did not receive (B identification string from zzz.18.53.102 Jul 28 07:00:07 (B magnus1 sshd[2568]: Did not receive identification string (B from zzz.155.91.132 Jul 29 05:59:55 magnus1 sshd[3798]: Did (B not receive identification string from zzz.235.37.77 Jul 30 (B 10:53:55 magnus1 sshd[5285]: Did not receive identification (B string from zzz.111.110.6 Jul 30 10:56:51 magnus1 (B sshd[5289]: (B Did not receive identification string from zzz.111.110.6 Jul (B (B 30 12:51:46 magnus1 sshd[5365]: Did not receive (B identification string from zzz.212.236.18 Jul 31 02:57:59 (B magnus1 sshd[5935]: Did not receive identification string (B from zzz.30.187.2 Aug 4 08:15:11 magnus1 sshd[14242]: Did (B not receive identification string from zzz.246.43.167 (B (B (B Previously, I have had easily two months of uptime on this (B server. (B (B Regards (B Magnus (B (B (B (B--- Steve Hovey [EMAIL PROTECTED] skrev: (BI would start with your cron jobs (B (B (BOn Thu, 14 Aug 2003, [iso-8859-1] Magnus J wrote: (B (B Hello everyone (B (B (B I'm not sure if I should have posted this to (Bfreebsd-security, (B but I start here. (B (B I'm out traveling, and finally got a chance to login to (B my server (B back home
Re: Server rebooted at 3 a.m. and 7 a.m. for the past few days
Hello Running /usr/local/etc/cvsup/update.sh manually caused the machine to reboot. Unfortunately, /var/log/cvsup.log doesn't provide any information about why. Any recommendation on what I should use to get more messages? Thanks Magnus --- Luke Kearney [EMAIL PROTECTED] skrev: - Original Message - From: Magnus J [EMAIL PROTECTED] To: Brent Wiese [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Sent: Friday, August 15, 2003 10:44 AM Subject: RE: Server rebooted at 3 a.m. and 7 a.m. for the past few days Hello dmesg shows no panic, and nothing that consumes much CPU has been running since the first reboot. Around 3 a.m. the daily periodic runs (which is default) and around 7 a.m. cvsup runs. Thanks Magnus --- Brent Wiese [EMAIL PROTECTED] skrev: Do you have any scripts that run at those times? If you run something like a database update or something that can crank some CPU cycles, you could be overheating the box, causing a reboot. Could happen all of a sudden if a fan decided to quit... Dmesg show any panics? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Magnus J Sent: Thursday, August 14, 2003 5:22 PM To: Steve Hovey Cc: [EMAIL PROTECTED] Subject: Re: Server rebooted at 3 a.m. and 7 a.m. for the past few days Hello Thanks for replying. /etc/crontab looks OK. This is how 'last' looks like (user1 is myself) user1 ttyp0zzz.12.28.40 Thu Aug 14 12:43 - 13:30 (00:46) user1 ttyp1zzz.12.28.40 Thu Aug 14 12:20 - 13:30 (01:09) user1 ttyp0zzz.12.28.40 Thu Aug 14 12:08 - 12:21 (00:12) user1 ttyp0zzz.12.27.12 Thu Aug 14 10:06 - 11:22 (01:15) user1 ttyp1zzz.12.28.52 Thu Aug 14 08:06 - 08:07 (00:00) user1 ttyp0zzz.12.28.52 Thu Aug 14 07:10 - 08:07 (00:56) reboot ~ Thu Aug 14 07:10 reboot ~ Thu Aug 14 03:09 reboot ~ Wed Aug 13 07:13 reboot ~ Wed Aug 13 03:09 reboot ~ Tue Aug 12 07:12 reboot ~ Tue Aug 12 03:09 reboot ~ Mon Aug 11 07:11 reboot ~ Mon Aug 11 03:09 reboot ~ Sun Aug 10 07:10 reboot ~ Sun Aug 10 03:08 reboot ~ Sat Aug 9 07:10 reboot ~ Sat Aug 9 04:22 reboot ~ Sat Aug 9 03:08 reboot ~ Fri Aug 8 07:10 reboot ~ Thu Aug 7 22:21 user1 ttyp4zzz.12.28.14 Mon Aug 4 22:39 - 22:40 (00:00) wtmp begins Mon Aug 4 22:39:55 CEST 2003 bash-2.05b# date Fri Aug 15 02:06:22 CEST 2003 bash-2.05b# Should I worry about these messages? Jul 16 14:06:47 magnus1 sshd[22292]: scanned from zzz.7.104.10 with SSH-1.0-SSH_ Version_Mapper. Don't panic. Jul 16 14:06:47 magnus1 sshd[22291]: Did not receive identification string from zzz.7.104.10 Jul 27 19:58:36 magnus1 sshd[1811]: scanned from zzz.18.53.102 with SSH-1.0-SSH_Ve Jul 27 19:58:36 magnus1 sshd[1811]: scanned from zzz.18.53.102 with SSH-1.0-SSH_Ve rsion_Mapper. Don't panic. Jul 27 19:58:36 magnus1 sshd[1810]: Did not receive identification string from zzz.18.53.102 Jul 28 07:00:07 magnus1 sshd[2568]: Did not receive identification string from zzz.155.91.132 Jul 29 05:59:55 magnus1 sshd[3798]: Did not receive identification string from zzz.235.37.77 Jul 30 10:53:55 magnus1 sshd[5285]: Did not receive identification string from zzz.111.110.6 Jul 30 10:56:51 magnus1 sshd[5289]: Did not receive identification string from zzz.111.110.6 Jul 30 12:51:46 magnus1 sshd[5365]: Did not receive identification string from zzz.212.236.18 Jul 31 02:57:59 magnus1 sshd[5935]: Did not receive identification string from zzz.30.187.2 Aug 4 08:15:11 magnus1 sshd[14242]: Did not receive identification string from zzz.246.43.167 Previously, I have had easily two months of uptime on this server. Regards Magnus --- Steve Hovey [EMAIL PROTECTED] skrev: I would start with your cron jobs On Thu, 14 Aug 2003, [iso-8859-1] Magnus J wrote: Hello everyone I'm not sure if I should have posted this to freebsd-security, but I start here. I'm out traveling, and finally got a chance
Re: Server rebooted at 3 a.m. and 7 a.m. for the past few days
On Fri, Aug 15, 2003 at 03:44:52AM +0200, Magnus J wrote:
Hello
dmesg shows no panic, and nothing that consumes much CPU has
been running since the first reboot.
Around 3 a.m. the daily periodic runs (which is default) and
around 7 a.m. cvsup runs.
How about taking cron down until you know it's not one of the cron jobs?
Recently I had to work out what was causing a server to fail for no
apparent reason at all - no messages to suggest anything going wrong
here either. In the end I used lsof:
/usr/ports/sysutils/lsof
to log all file activity, which turned up httpd as the cause of the
problem.
The script I used with lsof was adapted from the 'big_brother.perl5'
script that's included with the lsof dist which I'll include at the end
of this post. Be warned it eats up diskspace so make sure it logs to
somewhere with plenty of room and keep an eye on it regularly.
Script is pretty ugly with my changes and was only meant to be a quick hack...
#!/usr/bin/perl -w
#+##
# #
# File: big_brother.perl #
# #
# Description: check the network sockets with lsof to detect new connections #
# #
# Contributed by Lionel Cons [EMAIL PROTECTED] #
# #
#-##
# @(#)big_brother 1.12 08/14/96 Written by [EMAIL PROTECTED]
# no waranty! use this at your own risks!
#
# init setup
#
$verbose = 1;
#$lsof_opt = -itcp -iudp -Di -FcLPn -r 5;
#$lsof_opt = -r 5 -Di -FcLPn -n;
$lsof_opt = -r 5 -Di -n;
$pid_file = /var/run/big_brother.pid;
open(FD, $pid_file);
print FD $$;
close FD;
$SIG{'HUP'} = \hangup;
chop($hostname = `/bin/hostname`);
#$fq_hostname = (gethostbyname($hostname))[0];
$ymd=`date +%Y%m%d`;
chomp $ymd;
$lf=${hostname}.${ymd}.lsof.log;
# Set path to lsof.
$LSOF=/usr/local/sbin/lsof;
# Open logfile:
open_logfile;
#
# spy forever...
#
$| = 1;
die $LSOF is not executable\n unless -x $LSOF;
while (1) {
$lsof_pid = open(PIPE, $LSOF $lsof_opt 21 |)
|| die can't start $LSOF: $!\n;
print # , timestamp, $LSOF $lsof_opt, pid=$lsof_pid\n
if $verbose;
print #COMMAND PID USER P NAME\n;
$printed = $hanguped = $pid = $proto = 0;
while (PIPE) {
if(/^/){
print #ts: ,timestamp, \n;
} elsif (/^lsof: PID \d+, /) {
# fatal error message?
print *** $_;
last;
} elsif (/^lsof: /) {
# warning
warn * $_;
} else {
print;
}
}
kill('INT', $lsof_pid);
kill('KILL', $lsof_pid);
close(PIPE);
}
=comment
} elsif (/^p(\d+)$/) {
flush;
$pid = $1;
$proto = 0;
} elsif (/^c(.*)$/) {
$command = $1;
} elsif (/^L(.*)$/) {
$user = $1;
} elsif (/^P(.*)$/) {
flush;
$proto = $1;
} elsif (/^n(.*)$/) {
$name = $1;
# replace local hostname by 'localhost'
$name =~ s/\Q$fq_hostname\E/localhost/g;
$name =~ s/[0-9hms]+ ago//g;
} elsif (/^m$/) {
flush;
clean;
} else {
warn * bad output ignored: $_;
}
=cut
sub open_logfile {
open(FD, $lf);
select FD;
}
sub hangup {
#$hanguped = 1;
close FD;
$SIG{'HUP'} = \hangup;
open_logfile;
}
sub flush {
return unless $pid $proto;
return if skip;
$tag = sprintf(%-9s %5d %8s %1s %s, $command, $pid, $user,
substr($proto, 0, 1), $name);
unless (defined($seen{$tag})) {
print +$tag\n;
$printed++;
}
$seen{$tag} = 1;
}
sub clean {
my(@to_delete, $tag);
if ($hanguped) {
$hanguped = 0;
@to_delete = keys(%seen);
print # , timestamp, hangup received, rescanning all connections\n
if $verbose;
} else {
@to_delete = ();
foreach $tag (keys(%seen)) {
if ($seen{$tag} == 0) {
# not seen this time: delete it
push(@to_delete, $tag);
print -$tag\n;
$printed++;
} else {
# seen this time: reset the flag
$seen{$tag} = 0;
}
}
}
grep(delete($seen{$_}), @to_delete);
if ($printed 10) {
print # , timestamp, \n if $verbose;
$printed = 0;
RE: Server rebooted at 3 a.m. and 7 a.m. for the past few days
There are several system utils that'll stress the CPU/disk in the ports section. I'd try some of those to see if you can cause a reboot. If so, it might help diagnose... If you have a bad cpu fan, it doesn't take much to crash the box. I've seen this a lot in older dual p2/p3 box style cpus. The fan on the cpu who's less than a finger-width from the other CPU siezes up. The box will run fine under no load, but as soon as you put any kind of load on the box, it dies. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Magnus J Sent: Thursday, August 14, 2003 7:16 PM To: Luke Kearney Cc: [EMAIL PROTECTED] Subject: Re: Server rebooted at 3 a.m. and 7 a.m. for the past few days Hello Running /usr/local/etc/cvsup/update.sh manually caused the machine to reboot. Unfortunately, /var/log/cvsup.log doesn't provide any information about why. Any recommendation on what I should use to get more messages? Thanks Magnus --- Luke Kearney [EMAIL PROTECTED] skrev: - Original Message - From: Magnus J [EMAIL PROTECTED] To: Brent Wiese [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Sent: Friday, August 15, 2003 10:44 AM Subject: RE: Server rebooted at 3 a.m. and 7 a.m. for the past few days Hello dmesg shows no panic, and nothing that consumes much CPU has been running since the first reboot. Around 3 a.m. the daily periodic runs (which is default) and around 7 a.m. cvsup runs. Thanks Magnus --- Brent Wiese [EMAIL PROTECTED] skrev: Do you have any scripts that run at those times? If you run something like a database update or something that can crank some CPU cycles, you could be overheating the box, causing a reboot. Could happen all of a sudden if a fan decided to quit... Dmesg show any panics? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Magnus J Sent: Thursday, August 14, 2003 5:22 PM To: Steve Hovey Cc: [EMAIL PROTECTED] Subject: Re: Server rebooted at 3 a.m. and 7 a.m. for the past few days Hello Thanks for replying. /etc/crontab looks OK. This is how 'last' looks like (user1 is myself) user1 ttyp0zzz.12.28.40 Thu Aug 14 12:43 - 13:30 (00:46) user1 ttyp1zzz.12.28.40 Thu Aug 14 12:20 - 13:30 (01:09) user1 ttyp0zzz.12.28.40 Thu Aug 14 12:08 - 12:21 (00:12) user1 ttyp0zzz.12.27.12 Thu Aug 14 10:06 - 11:22 (01:15) user1 ttyp1zzz.12.28.52 Thu Aug 14 08:06 - 08:07 (00:00) user1 ttyp0zzz.12.28.52 Thu Aug 14 07:10 - 08:07 (00:56) reboot ~ Thu Aug 14 07:10 reboot ~ Thu Aug 14 03:09 reboot ~ Wed Aug 13 07:13 reboot ~ Wed Aug 13 03:09 reboot ~ Tue Aug 12 07:12 reboot ~ Tue Aug 12 03:09 reboot ~ Mon Aug 11 07:11 reboot ~ Mon Aug 11 03:09 reboot ~ Sun Aug 10 07:10 reboot ~ Sun Aug 10 03:08 reboot ~ Sat Aug 9 07:10 reboot ~ Sat Aug 9 04:22 reboot ~ Sat Aug 9 03:08 reboot ~ Fri Aug 8 07:10 reboot ~ Thu Aug 7 22:21 user1 ttyp4zzz.12.28.14 Mon Aug 4 22:39 - 22:40 (00:00) wtmp begins Mon Aug 4 22:39:55 CEST 2003 bash-2.05b# date Fri Aug 15 02:06:22 CEST 2003 bash-2.05b# Should I worry about these messages? Jul 16 14:06:47 magnus1 sshd[22292]: scanned from zzz.7.104.10 with SSH-1.0-SSH_ Version_Mapper. Don't panic. Jul 16 14:06:47 magnus1 sshd[22291]: Did not receive identification string from zzz.7.104.10 Jul 27 19:58:36 magnus1 sshd[1811]: scanned from zzz.18.53.102 with SSH-1.0-SSH_Ve Jul 27 19:58:36 magnus1 sshd[1811]: scanned from zzz.18.53.102 with SSH-1.0-SSH_Ve rsion_Mapper. Don't panic. Jul 27 19:58:36 magnus1 sshd[1810]: Did not receive identification string from zzz.18.53.102 Jul 28 07:00:07 magnus1 sshd[2568]: Did not receive identification string from zzz.155.91.132 Jul 29 05:59:55 magnus1 sshd[3798]: Did not receive identification string from zzz.235.37.77 Jul 30 10:53:55 magnus1 sshd[5285]: Did not receive
