Never seen this before -- all daemons and all user processes killed. Zap. It
happened around 23:17 Sunday, Chicago time (that's when /var/log/* abruptly
stopped). Any idea what might cause this?
I was ssh'd in to my Debian server and... disconnected. No problem, I was
using *screen* to *vim* some *Catalyst* modules, so I'll just reconnect and
reattach... connection refused.
Wha?
Tried telnet to port 22, no sign of life. Tried telnet to port 80, no sign
of life.
Went to the server room, logged in on the console:
will@darth:~$ uptime
23:58:11 *up 583 days*, 3:03, 6 users, load average: 0.00, 0.02, 0.08
So the server hadn't had a hard reset, still up 583 days. In /var/log/syslog
there are the usual cron logs up to about 23:17 and then.. nothing.
will@darth:~$ ps afx
PID TTY STAT TIME COMMAND
2 ? S< 0:00 [kthreadd]
3 ? S< 1:13 \_ [migration/0]
4 ? S< 29:21 \_ [ksoftirqd/0]
5 ? S< 0:32 \_ [watchdog/0]
6 ? S< 1:12 \_ [migration/1]
7 ? S< 77:19 \_ [ksoftirqd/1]
8 ? S< 0:02 \_ [watchdog/1]
9 ? S< 44:52 \_ [events/0]
10 ? S< 78:24 \_ [events/1]
11 ? S< 0:00 \_ [khelper]
44 ? S< 13:20 \_ [kblockd/0]
45 ? S< 0:40 \_ [kblockd/1]
47 ? S< 0:00 \_ [kacpid]
48 ? S< 0:00 \_ [kacpi_notify]
121 ? S< 0:00 \_ [kseriod]
161 ? S< 19:53 \_ [kswapd0]
162 ? S< 0:00 \_ [aio/0]
163 ? S< 0:00 \_ [aio/1]
642 ? S< 0:00 \_ [ksuspend_usbd]
647 ? S< 0:00 \_ [khubd]
761 ? S< 0:00 \_ [ata/0]
764 ? S< 0:00 \_ [ata/1]
765 ? S< 0:00 \_ [ata_aux]
774 ? S< 0:00 \_ [scsi_eh_0]
775 ? S< 0:00 \_ [scsi_eh_1]
877 ? S< 42:46 \_ [kjournald]
1301 ? S< 17:22 \_ [edac-poller]
1384 ? S< 0:00 \_ [kpsmoused]
1640 ? S< 0:00 \_ [kstriped]
1654 ? S< 0:00 \_ [ksnapd]
1681 ? S< 76:13 \_ [kjournald]
1682 ? S< 126:18 \_ [kjournald]
12642 ? S 0:09 \_ [pdflush]
19987 ? S 0:00 \_ [pdflush]
1 ? Ss 10:04 init [2]
11064 tty2 Ss+ 0:00 /sbin/getty 38400 tty2
11065 tty3 Ss+ 0:00 /sbin/getty 38400 tty3
11066 tty4 Ss+ 0:00 /sbin/getty 38400 tty4
11067 tty5 Ss+ 0:00 /sbin/getty 38400 tty5
11068 tty6 Ss+ 0:00 /sbin/getty 38400 tty6
12995 tty1 Ss 0:00 /bin/login --
13077 tty1 S 0:00 \_ -bash
13107 tty1 R+ 0:00 \_ ps afx
Freaky: init, that's process #1, isn't at the top? And all daemons except
for getty were gone. All user processes including my screen sessions! and
vim sessions!, were gone.
Checking 'last' didn't show any suspicious activity.
In kern.log there's only
Jan 23 23:04:59 darth kernel: [64084756.601774] exploit[25161]: segfault at
10c00b ip 00000000 sp deadc01d error 6
Jan 23 23:05:08 darth kernel: [64084765.528734] NET: Registered protocol
family 5
After a quick
$ sudo bash
# cd /etc/rc2.d
# for x in S*; do sh $x start; done
the server was back up and serving... and then the saddest sight of all, of
course:
will@darth:~$ screen -ls
There is a screen on:
26279.pts-3.darth (06/19/09 21:54:31) (Dead ???)
Remove dead screens with 'screen -wipe'.
1 Socket in /var/run/screen/S-will.
:(
$ tail /var/log/messages
Jan 23 22:56:26 darth -- MARK --
Jan 23 23:04:59 darth kernel: [64084756.601774] exploit[25161]: segfault at
10c00b ip 00000000 sp deadc01d error 6
Jan 23 23:05:08 darth kernel: [64084765.528734] NET: Registered protocol
family 5
Jan 23 23:16:26 darth -- MARK --
Jan 23 23:47:02 darth syslogd 1.5.0#5: restart.
So everything crapped out after 23:16, and I restarted it at 23:47.
*Anybody got a clue as to what might have happened to kill all daemons and
user-processes in one swoop? This has been a rock-solid Debian server for
years...*
will@darth:~$ cat /etc/debian_version
5.0.4
--
The first step towards getting somewhere is to decide that you are not going
to stay where you are. -- J.P.Morgan
