So the guys was either running unknown binaries as root, compiling
unknown source as root or running some daemon as root, so what? Every
newbie knows that doing so is simply asking for trouble, even though
most do it. A production server shouldn't anyway be running unknown
software.
On Wed, 2003-08-13 at 16:54, Carl Cerecke wrote:
> This was forwarded to me. The guy works for a NZ company.
> Names have been genericised [not deprec(i)ated] :-)
>
> Cheers,
> Carl
>
> I thought you guys might get a sadistic laugh out of the fun-and-games
> we've been going through at <company> recently. If anyone ever suggests
> that linux is somehow less susceptible to viruses, worms and the like,
> hit them.
>
> It all started when we suddenly couldn't log in on the console.
> Fortunately ssh worked, and I quickly established that /bin/login was
> zero-length. Not suspecting human malice at this stage I tried to copy
> /bin/login from another machine, only to be told "Operation Not
> Permitted". Huh? The kernel not letting me overwrite a file even as
> root? So then I had to learn about EXT2 filesystem attributes, which I'd
> never heard of before. It turns out that there is an "immutable"
> attribute, which means that not even root can overwrite the file. I
> found this out via a google search - which returned a Russian hackers
> page. I couldn't read the Russian, but I could see the commands they
> were using. The commands for checking/setting EXT2 attributes are lsattr
> and chattr, but when I tried to run either of these I got a seg fault!
> Fortunately I had an identically-configured machine on site, so I just
> copied lsattr/chattr over, and they worked. But not for long.
>
> By this stage I was pretty sure that I was dealing with a worm or virus.
> This was confirmed when I noticed that a lot of the standard system
> binaries kept changing size. Eventually I worked out that most of the
> binaries (ls, cp, touch, awk, and so on) had been hacked such that each
> time they were run they made sure that all the other binaries were
> infected and then did what they were supposed to do. In time I also
> discovered that the hacked copies of top, ps, ls, and netstat were also
> hiding the worm's processes and files. So I would copy uncorrupted
> binaries from the other machine, but every now-and-then I would run a
> command that I didn't know was hacked and the whole system would be
> infected again.
>
> Then I discovered that the ssh/scp binaries had been replaced by
> entirely new ones. These worked fine, but were presumably doing
> something else as well (back doors? password grabbers?). The directory
> in which these new versions were compiled up was called
> "/var/lib/games/.. " (note the terminal space). I think this was done so
> that you would be tempted to type "rm -rf .. ", without escaping the
> space on the end.
>
> Then I discovered that whenever I restarted the web server some other
> process started up and grabbed port 80, even though there was nothing
> wrong with the startup files and I was using an unhacked version of
> httpd. Eventually I discovered that /etc/ld.so.cache had been replaced,
> and I suspect that it provided a hacked version of some standard routine
> that httpd calls on start-up and which actually launched some unknown
> process. What this process was doing with port 80 I don't know. I wish I
> had visited our web sites while this other process was running - I
> suspect it was showing a "Na-na-na-na-na message."
>
> If you're wondering why I didn't just reinstall the operating system
> from scratch, well, this was all happening on <company>'s main, live,
> 24/7 production server supplying NAT, DNS, email, and half the web
> sites. I was fixing all this under intense pressure as clients from all
> over the world were ringing to complain that things weren't working.
>
> The final insult came several days later, after I had thought that I had
> finally beaten the worm dead. The server got rebooted, and I got hauled
> in to work because it didn't come up again. The kernel was running, but
> no services (including getty and ssh) were running. Fortunately the old
> lilo trick ("linux -init=/bin/bash") worked, so I could at least get on.
> I discovered that the toerags had actually replaced init itself, so on a
> reboot the kernel ran init which did - who knows what. It was running
> and doing something, but it certainly wasn't doing what well-bred inits
> are supposed to do (like letting people log in for instance).
>
