It's almost always a kiddie in Romania (or one of the Baltic Republics). There are neighborhood competitions there to 'out hack' each other, to the point that they put the number of US servers they've pwn3d on big signs in their living room windows.
I started putting the following lines on every Linux box we install a while back: iptables -A INPUT -p tcp -m tcp --dport 21 -m state --state NEW -j FTPSCAN iptables -A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -j SSHSCAN iptables -A FTPSCAN -m recent --set --name FTP --rsource iptables -A FTPSCAN -m recent --update --seconds 300 --hitcount 5 --name FTP --rsource -j LOG --log-prefix "FTP Scan blocked: " --log-level 6 iptables -A FTPSCAN -m recent --update --seconds 300 --hitcount 5 --name FTP --rsource -j DROP iptables -A SSHSCAN -m recent --set --name SSH --rsource iptables -A SSHSCAN -m recent --update --seconds 300 --hitcount 5 --name SSH --rsource -j LOG --log-prefix "SSH Scan blocked: " --log-level 6 iptables -A SSHSCAN -m recent --update --seconds 300 --hitcount 5 --name SSH --rsource -j DROP Every once in a blue moon it catches a legit user, but then it times out after 5 minutes and their access is restored. However on the positive side, it cut down my logwatch emails from megabytes to a single page on nearly every server we put it on, because it does completely automatic storm control and recovery. I have a couple FWBuilder lines that accomplish the same thing if'n anyone's interested as well. R -----Original Message----- From: joe golden [mailto:[email protected]] Sent: Thursday, May 12, 2011 9:34 PM To: [email protected] Subject: Re: Hacked Server (no, really!) Patrick, check your logs: therin's the trail to catch the thief ;-) If user nagios logged in it should be recorded in /var/log/auth.log. grep nagios /var/log/auth.log. Also checked compressed auth.logs. If someone accessed something via the web, there might be something in the apache logs. If that someone was acting as user nagios, grep nagios /var/log/apache2/access.log. If they might have caused an error, grep nagios /var/log/apache2/error.log. As above check the compressed logs too. If there's a URL you suspect had something to do with it, grep URL_snippet /var/log/apache2/access.log. I run a Debian server that's online 24/7 with a highly attractive URL (that won't be mentioned here ;-) ). I always put ssh on a non standard port, otherwise everybody and his brother are knocking on port 22. Any chance you went without a condom and ftped into the box at some point? Somebody could have sniffed a password long ago. Maybe. -- Joe Golden /_\ www.Triangul.us /_\ websites with class On 05/12/2011 06:26 PM, Jim Lawson wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > > On 05/12/2011 05:18 PM, Patrick Litke wrote: >> 1) How did they gain access? I doubt that it would be possible for them >> to bruteforce that password - it was 10 characters long, upper and lower >> case, numbers and symbols. So then what vulnerability did they exploit >> to get in? > > It does seem unlikely that they brute forced the account's password[1]. > More likely they got in through a problem with your webserver > configuration. Is it possible that the "nagios" user has write > permission to some directory, which the webserver or nagios is allowed > to execute as well? That's a common method of penetration - have nagios > (or whatever webapp) write out a shellcode somewhere, and then get the > webserver to execute that shellcode as a CGI. Once you're that far, use > the shellcode to update nagios's $HOME/.ssh/authorized_keys file and > drop in your own SSH key, so you can get a regular SSH session. > >> 2) What else do I need to do to secure a Debian install for dedicated >> online time plugged into the ebil interwebs? > > There is no easy answer... It sounds to me like you have a lot of web > services running (you list "full lamp stack", nagios, webmin, samba, and > others), and one of those services had a hole. Since it appears they got > into the nagios account, you should look at what that user runs. Is it > just nagios, or does Apache run as the nagios user? > > Jim > > [1] I have to ask, why does the nagios user have a password? It seems to > me that the password field for that account should be "*", meaning > password authentication isn't possible ... If you need to do work as > nagios, you can become root and use 'su'. > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (GNU/Linux) > > iD8DBQFNzF6K2USjQZu2vYURAj5ZAJ43gvXVk7IXAgkomYMFD1PyBP8teACgtrrC > ybjkX3jVjaoU3NvJRUpbYi8= > =RdJn > -----END PGP SIGNATURE----- >
