Patrick, given the situation the full reinstall from scratch is
certainly your best bet. I've done recovery of hacked servers before
and it is never easy because the hackers tricks are so clever.
For those who are not aware here are some key things that I've
learned about hacked systems.
1. Don't trust the logs. Many rootkits will modify the log files to
remove all evidence that they were active. A log file whose entries
suddenly stop is usually good evidence of a breakin. If you have
logwatch working be sure to have the emails sent to a destination
that is not the server that they are reporting upon. That way you
have at least a small chance that the email goes out before the log
files are modified.
2. Don't trust the utilities that you normally would use for
detecting problems with the system such as ls, netstat, vmstat, who,
whois, find, lsof, and so forth. It is common for all of these to be
replaced with bogus versions that are coded so that they explicitly
hide directories, files, logged in users, etc. Boot off a Linux
CD/DVD and use those binaries for your investigation.
3. Sometimes you can find evidence of hacking by looking for oddball
directories. My favorite one was named ... and was easy to overlook
while scanning directory listings. Yes that's three dots rather than
two. Totally legitimate directory name but not one that you'll ever
find on a normal Linux system.
4. Keep your iptables tight. Don't open ports that are not needed.
5. Watch the automatically created accounts and disable those that
you'll never actually login to. Such as that nagios account. A shell
of /bin/false will prevent any kind of external login whether using
ssh keys or passwords.
6. If you get hacked the very best thing to do is to rebuild your
system from scratch. If you have to move files from the old system
to the new then you must be very very careful to not carry along any
hidden or obscure files that don't belong with your original files.
I hope this helps.
Dan
On 5/12/11 5:18 PM, Patrick Litke wrote:
Hello, Hello all! This is my first post to the list
serv, after much encouragement from David Tisdell
I've taken a few classes with Dave covering Network Security and
Linux, and have been playing around with Linux for a few years
now. I wound up buying a set of cheap hardware for a linux server
(both my self and a friend did the same) to be predominantly used
as a VoD box / media server / Samba server. I'm planning on using
mine as a test bed for software development, and subsequently need
it connected to the internet.
We wound up using his server as a test box for everything. We got
Debian installed, full lamp stack, nagios, webmin, samba, and a
few others. Everything was working fine - we had remote SSH
access, full http access (only over SSL though), remote access to
webmin - the works. We did not have pasword-less logins (ie:
using keys) for SSH logins though.
A few days went by, and we got a phone call from ComCast about a
report they got from a company (whom they wouldn't name) basically
saying that they were being FTP attacked from our IP address. So
we pulled the server off line, ran a rootkit scan which came back
clean, did a little more tweaking, took a look at our iptables
config and our router set up (80, 443, 22, 125(3?), 10000 were the
ports we had forwarded) and everything seemed ok. So we put it
back online. The next day, we reviewed the router logs and there
were thousands of STORM ATTACK entries (we were being ddos'd?) and
several unique IP's that were foreign to us. We did a few
lookups/traces...
pug@jabberwocky:~$ whois -H --verbose 79.115.177.0
Using server whois.ripe.net.
Query string: "-V Md5.0 79.115.177.0"
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf
% Note: this output has been filtered.
% To receive output for a database update, use the "-B"
flag.
% Information related to '79.115.132.0 - 79.115.187.255'
inetnum: 79.115.132.0 - 79.115.187.255
netname: RO-RCS-RDS-FIBERLINK
descr: RCS & RDS S.A.
descr: FiberLink Customers
descr: Bucuresti
country: RO
admin-c: RDS-RIPE
tech-c: RDS-RIPE
status: ASSIGNED PA
mnt-by: AS8708-MNT
source: RIPE # Filtered
role: Romania Data Systems NOC
address: 71-75 Dr. Staicovici
address: Bucharest / ROMANIA
phone: +40 21 30 10 888
fax-no: +40 21 30 10 892
abuse-mailbox: [email protected]
admin-c: CN19-RIPE
admin-c: VIG10-RIPE
tech-c: CN19-RIPE
tech-c: VIG10-RIPE
nic-hdl: RDS-RIPE
mnt-by: AS8708-MNT
remarks:
+--------------------------------------------------------------+
remarks: | ABUSE CONTACT: [email protected]
IN CASE OF HACK ATTACKS, |
remarks: | ILLEGAL ACTIVITY, VIOLATION, SCANS,
PROBES, SPAM, ETC. |
remarks: | !! PLEASE DO NOT CONTACT OTHER PERSONS FOR
THESE PROBLEMS !! |
remarks:
+--------------------------------------------------------------+
source: RIPE # Filtered
% Information related to '79.112.0.0/13AS8708'
route: 79.112.0.0/13
descr: RDSNET
origin: AS8708
mnt-by: AS8708-MNT
source: RIPE # Filtered
And our traceroute (from a different IP out of our routers
logs)...
slippy@Slippy:~$ traceroute 79.115.187.255
traceroute to 79.115.187.255 (79.115.187.255), 30 hops max, 60
byte packets
1 192.168.1.1 (192.168.1.1) 2.476 ms 7.208 ms 7.600 ms
2 10.19.14.1 (10.19.14.1) 27.978 ms 29.492 ms 30.777 ms
3 static-64-222-84-42.burl.east.myfairpoint.net
(64.222.84.42) 32.944 ms 34.313 ms 35.889 ms
4 POS4-0-1.GW16.NYC9.ALTER.NET
(208.192.176.205) 51.879 ms 53.312 ms 54.770 ms
5 0.ge-3-0-0.XT1.NYC9.ALTER.NET
(152.63.22.138) 56.072 ms 57.433 ms 58.874 ms
6 0.so-6-0-2.XL3.NYC4.ALTER.NET
(152.63.10.21) 60.336 ms 40.160 ms 67.725 ms
7 GigabitEthernet4-0-0.GW1.NYC4.ALTER.NET
(152.63.20.97) 43.537 ms GigabitEthernet6-0-0.GW1.NYC4.ALTER.NET
(152.63.20.49) 45.893 ms GigabitEthernet4-0-0.GW1.NYC4.ALTER.NET
(152.63.20.97) 46.337 ms
8 teliasonera-gw.customer.alter.net
(157.130.60.14) 49.415 ms 49.822 ms 53.146 ms
9 nyk-bb2-link.telia.net
(213.155.130.244) 68.458 ms 70.546 ms 70.952 ms
10 kbn-bb2-link.telia.net
(80.91.247.120) 226.735 ms 239.932 ms 240.841 ms
11 hbg-bb2-link.telia.net
(80.91.249.207) 168.734 ms 170.594 ms 171.583 ms
12 win-bb2-link.telia.net
(80.91.246.25) 149.003 ms win-bb2-link.telia.net
(213.155.131.111) 151.061 ms win-bb2-link.telia.net
(80.91.246.25) 154.174 ms
13 buca-b1-link.telia.net
(80.91.246.155) 192.605 ms 193.127 ms buca-b1-link.telia.net (80.91.247.81)
174.068 ms
14 rcsrds-ic-141235-buca-b1.c.telia.net
(213.248.83.170) 176.061 ms 176.270 ms 177.728 ms
15 213-154-124-95.rdsnet.ro
(213.154.124.95) 185.042 ms 186.032 ms 186.439 ms
Our Routers logs have thousands of entries akin to the following
[DoS attack: STORM] attack packets in last 20 sec from ip
[213.161.192.51], Tuesday, May 03,2011 11:29:17
[DoS attack: Smurf] attack packets in last 20 sec from ip
[82.176.5.255], Tuesday, May 03,2011 11:28:59
Naturally, we emailed their ISP reporting the incident - but it's
romania, who cares?
At any rate, this is where we began to scratch our heads. A day
or so later, I was logged in remotely and just to see if I could
see Dave logged in, I ran a
pug@jabberwocky:~$ who -a
2011-05-04 09:12 415
id=si term=0 exit=0
system boot 2011-05-04 09:12
run-level 2 2011-05-04 09:12
last=S
2011-05-04 09:12 1048
id=l2 term=0 exit=0
LOGIN tty1 2011-05-04 09:12 1876
id=1
LOGIN tty6 2011-05-04 09:12 1881
id=6
LOGIN tty3 2011-05-04 09:12 1878
id=3
LOGIN tty4 2011-05-04 09:12 1879
id=4
LOGIN tty5 2011-05-04 09:12 1880
id=5
LOGIN tty2 2011-05-04 09:12 1877
id=2
nagios + pts/0 2011-05-04 15:19 00:06 14217
(79.115.177.0)
pug + pts/1 2011-05-04 12:40 00:16 7866 ( pool-72-92-154-71.burl.east.myfairpoint.net)
pug + pts/2 2011-05-04 15:29 . 15633 ( pool-72-92-154-71.burl.east.myfairpoint.net)
pug@jabberwocky:~$
and saw our local nagios user logged in from that IP
address. So naturally, we kicked him out.
After that, we KNEW we had unauthorized remote access so we took
the server down again. At this point, we called up our local
comcast techie (the guy that called us to inform us what was
happening) and he said that he had the same issue with his home
server. He wound up banning the African, Asian and South
American continents with iptables - which is something that I
would like to do.
But this left us with questions.
1) How did they gain access? I doubt that it would be possible
for them to bruteforce that password - it was 10 characters
long, upper and lower case, numbers and symbols. So then what
vulnerability did they exploit to get in?
2) What else do I need to do to secure a Debian install for
dedicated online time plugged into the ebil interwebs?
We now are starting over, on both servers. We have fresh Debian
installs (no gui - why bother?). The list of things to do that
I have come up with is as follows
- Install Linux
- Install Snort-mysql and acidbase w/ addons & configure
with/for IPTables
- Install Nagios (is there some vulnerability to this?)
- Install Webmin
- Install Cactus
- Ensure that SSH is listening/accepting logins with keys
ONLY (it is, now)
This is all before we do anything else - our LAMP stack will be
installed at the end anyway having installed everything with
snort. Is there anything else that we should be taking in to
consideration / that we should do? And, having played around
with a snort install... I am WAY over my head, I have no idea
how the classes of IP blocks break down, or how to set SNORT to
listen on a single adapter (as the router only forwards ports to
the server, nowhere else on the network).
I'm looking forward to the next meeting as I am thinking about
showing up! :)
|
