This was forwarded to me. The guy works for a NZ company. Names have been genericised [not deprec(i)ated] :-)
Cheers, Carl
I thought you guys might get a sadistic laugh out of the fun-and-games
we've been going through at <company> recently. If anyone ever suggests that linux is somehow less susceptible to viruses, worms and the like, hit them.
It all started when we suddenly couldn't log in on the console. Fortunately ssh worked, and I quickly established that /bin/login was zero-length. Not suspecting human malice at this stage I tried to copy /bin/login from another machine, only to be told "Operation Not Permitted". Huh? The kernel not letting me overwrite a file even as root? So then I had to learn about EXT2 filesystem attributes, which I'd never heard of before. It turns out that there is an "immutable" attribute, which means that not even root can overwrite the file. I found this out via a google search - which returned a Russian hackers page. I couldn't read the Russian, but I could see the commands they were using. The commands for checking/setting EXT2 attributes are lsattr and chattr, but when I tried to run either of these I got a seg fault! Fortunately I had an identically-configured machine on site, so I just copied lsattr/chattr over, and they worked. But not for long.
By this stage I was pretty sure that I was dealing with a worm or virus. This was confirmed when I noticed that a lot of the standard system binaries kept changing size. Eventually I worked out that most of the binaries (ls, cp, touch, awk, and so on) had been hacked such that each time they were run they made sure that all the other binaries were infected and then did what they were supposed to do. In time I also discovered that the hacked copies of top, ps, ls, and netstat were also hiding the worm's processes and files. So I would copy uncorrupted binaries from the other machine, but every now-and-then I would run a command that I didn't know was hacked and the whole system would be infected again.
Then I discovered that the ssh/scp binaries had been replaced by entirely new ones. These worked fine, but were presumably doing something else as well (back doors? password grabbers?). The directory in which these new versions were compiled up was called "/var/lib/games/.. " (note the terminal space). I think this was done so that you would be tempted to type "rm -rf .. ", without escaping the space on the end.
Then I discovered that whenever I restarted the web server some other process started up and grabbed port 80, even though there was nothing wrong with the startup files and I was using an unhacked version of httpd. Eventually I discovered that /etc/ld.so.cache had been replaced, and I suspect that it provided a hacked version of some standard routine that httpd calls on start-up and which actually launched some unknown process. What this process was doing with port 80 I don't know. I wish I had visited our web sites while this other process was running - I suspect it was showing a "Na-na-na-na-na message."
If you're wondering why I didn't just reinstall the operating system
from scratch, well, this was all happening on <company>'s main, live, 24/7 production server supplying NAT, DNS, email, and half the web sites. I was fixing all this under intense pressure as clients from all over the world were ringing to complain that things weren't working.
The final insult came several days later, after I had thought that I had
finally beaten the worm dead. The server got rebooted, and I got hauled
in to work because it didn't come up again. The kernel was running, but
no services (including getty and ssh) were running. Fortunately the old
lilo trick ("linux -init=/bin/bash") worked, so I could at least get on.
I discovered that the toerags had actually replaced init itself, so on a
reboot the kernel ran init which did - who knows what. It was running
and doing something, but it certainly wasn't doing what well-bred inits
are supposed to do (like letting people log in for instance).