Check the first IP in your logs where someone hits up check.cgi 213.169.63.254 is an IP in Bulgaria. Do you have any customers with a business address from Bulgaria?
Also, the octet .63 is often at the tail end of a network range, and .254 is often the last address of a range. It's possible that this is a WAN router for a larger network or something and I would bet that some unauthorized person is doing something undesirable from that networks' perspective as well. check IP's with "whois -h whois.arin.net 213.169.63.254" Jeff Lasman wrote: > On Tuesday 25 November 2008 01:18 pm, David Kaiser wrote: > > >> Well, it is possible that some process was able to change it's >> process info to read "/usr/bin/perl -w./check.cgi" when there was no >> check.cgi file anywhere. Just a decoy trick from a bad process. >> > > Exactly what we found ... Thanks for bringing this to my attention. > > >> (Just thinking out loud...) could the 'check.cgi' file live inside a >> loopback filesystem, which was somehow unmounted after the process >> started? Do you see any funny entries in the 'mount' output? See >> any funny files in global-writable areas, like /tmp ? >> > > No, we never found the location of the file, but the processes were > running on the server and an hour ago when I started looking again they > were taking up about 95% of cpu resources as shown in top by %CPU. > > >> What do you get from (if the pid's are still valid): >> "ls -l /proc/28720/cwd" >> "ls -l /proc/28720/exe" >> "cat /proc/28720/environ" >> "cat /proc/28720/mountinfo" >> "cat /proc/28720/maps" >> and "cat /proc/28720/cmdline" >> > > The important ones were environ and cmdline. > > Five of them running, and using 95% of cpu resources. > > The environ line looked like this: > > TERM=xtermPATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/X11R6/bin_=./check.cgi > PWD=/home/flexigen/domains/spijtag.com/public_html > LANG=en_US.UTF-8SHLVL=2 > > (I've put in line breaks) > > and the cmdline looked like this: > > /usr/bin/perl-w./check.cgi > > on all five of them. > > I've suspended the account and notified the user (who it turns out > hasn't paid in over a year anyway). I also manually shut down the five > processes by PID. > > Now to wait and see if they come back. > > I found a public_html/scripts directory with a timestamp with today's > date, and no files with today's date in it. My guess is the file is > being downloaded and run, and then deleted to make it hard to find. > > I found in some zipped up logs our system creates for individual sites, > from the 23rd, the following: > > <snip> > # grep check.cgi spijtag.com.* > spijtag.com.error.log.1:[Sun Nov 23 08:31:11 2008] [error] [client > 213.169.63.254] Options ExecCGI is off in this > directory: /home/flexigen/domains/spijtag.com/public_html/scripts/check.cgi > spijtag.com.error.log.1:[Sun Nov 23 09:26:38 2008] [error] [client > 69.65.10.202] Options ExecCGI is off in this > directory: /home/flexigen/domains/spijtag.com/public_html/scripts/check.cgi > spijtag.com.error.log.1:[Sun Nov 23 10:14:08 2008] [error] [client > 69.65.10.202] Options ExecCGI is off in this > directory: /home/flexigen/domains/spijtag.com/public_html/scripts/check.cgi > spijtag.com.error.log.1:[Sun Nov 23 12:04:32 2008] [error] [client > 69.65.10.202] Options ExecCGI is off in this > directory: /home/flexigen/domains/spijtag.com/public_html/scripts/check.cgi > spijtag.com.error.log.1:[Sun Nov 23 13:20:36 2008] [error] [client > 69.65.10.202] Options ExecCGI is off in this > directory: /home/flexigen/domains/spijtag.com/public_html/scripts/check.cgi > spijtag.com.error.log.1:[Sun Nov 23 14:02:04 2008] [error] [client > 69.65.10.202] Options ExecCGI is off in this > directory: /home/flexigen/domains/spijtag.com/public_html/scripts/check.cgi > spijtag.com.log.1:213.169.63.254 - - [23/Nov/2008:08:31:11 -0800] > "GET /scripts/check.cgi HTTP/1.0" 403 - "-" "Mozilla/4.0 (compatible; > MSIE 6.0; Windows NT 5.1; SV1)" > spijtag.com.log.1:69.65.10.202 - - [23/Nov/2008:09:26:38 -0800] > "GET /scripts/check.cgi HTTP/1.0" 403 - "-" "Mozilla/4.0 (compatible; > MSIE 6.0; Windows NT 5.1; SV1)" > spijtag.com.log.1:69.65.10.202 - - [23/Nov/2008:10:14:08 -0800] > "GET /scripts/check.cgi HTTP/1.0" 403 - "-" "Mozilla/4.0 (compatible; > MSIE 6.0; Windows NT 5.1; SV1)" > spijtag.com.log.1:69.65.10.202 - - [23/Nov/2008:12:04:32 -0800] > "GET /scripts/check.cgi HTTP/1.0" 403 - "-" "Mozilla/4.0 (compatible; > MSIE 6.0; Windows NT 5.1; SV1)" > spijtag.com.log.1:69.65.10.202 - - [23/Nov/2008:13:20:36 -0800] > "GET /scripts/check.cgi HTTP/1.0" 403 - "-" "Mozilla/4.0 (compatible; > MSIE 6.0; Windows NT 5.1; SV1)" > spijtag.com.log.1:69.65.10.202 - - [23/Nov/2008:14:02:04 -0800] > "GET /scripts/check.cgi HTTP/1.0" 403 - "-" "Mozilla/4.0 (compatible; > MSIE 6.0; Windows NT 5.1; SV1)" > # > </snip> > > >From the above the errors from the errors from the error log, the good > accesses from the access log. > > I don't know yet if this was the source of the spam or not. > > Here's what I did about the spam: > > I reconfiged our squirrelmail webmail file to use the sendmail link > rather than smtp to port 25 to send email, and I blocked all email from > port 127.0.0.1. Since then the emails are being blocked by my local > blocklist. There have been no mails since I suspended the site, but > I'll check again in the morning to see if that's resolved the issue. > > Again, thanks for your help. > > Jeff >
