On Tuesday 25 November 2008 01:18 pm, David Kaiser wrote: > Well, it is possible that some process was able to change it's > process info to read "/usr/bin/perl -w./check.cgi" when there was no > check.cgi file anywhere. Just a decoy trick from a bad process.
Exactly what we found ... Thanks for bringing this to my attention. > (Just thinking out loud...) could the 'check.cgi' file live inside a > loopback filesystem, which was somehow unmounted after the process > started? Do you see any funny entries in the 'mount' output? See > any funny files in global-writable areas, like /tmp ? No, we never found the location of the file, but the processes were running on the server and an hour ago when I started looking again they were taking up about 95% of cpu resources as shown in top by %CPU. > What do you get from (if the pid's are still valid): > "ls -l /proc/28720/cwd" > "ls -l /proc/28720/exe" > "cat /proc/28720/environ" > "cat /proc/28720/mountinfo" > "cat /proc/28720/maps" > and "cat /proc/28720/cmdline" The important ones were environ and cmdline. Five of them running, and using 95% of cpu resources. The environ line looked like this: TERM=xtermPATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/X11R6/bin_=./check.cgi PWD=/home/flexigen/domains/spijtag.com/public_html LANG=en_US.UTF-8SHLVL=2 (I've put in line breaks) and the cmdline looked like this: /usr/bin/perl-w./check.cgi on all five of them. I've suspended the account and notified the user (who it turns out hasn't paid in over a year anyway). I also manually shut down the five processes by PID. Now to wait and see if they come back. I found a public_html/scripts directory with a timestamp with today's date, and no files with today's date in it. My guess is the file is being downloaded and run, and then deleted to make it hard to find. I found in some zipped up logs our system creates for individual sites, from the 23rd, the following: <snip> # grep check.cgi spijtag.com.* spijtag.com.error.log.1:[Sun Nov 23 08:31:11 2008] [error] [client 213.169.63.254] Options ExecCGI is off in this directory: /home/flexigen/domains/spijtag.com/public_html/scripts/check.cgi spijtag.com.error.log.1:[Sun Nov 23 09:26:38 2008] [error] [client 69.65.10.202] Options ExecCGI is off in this directory: /home/flexigen/domains/spijtag.com/public_html/scripts/check.cgi spijtag.com.error.log.1:[Sun Nov 23 10:14:08 2008] [error] [client 69.65.10.202] Options ExecCGI is off in this directory: /home/flexigen/domains/spijtag.com/public_html/scripts/check.cgi spijtag.com.error.log.1:[Sun Nov 23 12:04:32 2008] [error] [client 69.65.10.202] Options ExecCGI is off in this directory: /home/flexigen/domains/spijtag.com/public_html/scripts/check.cgi spijtag.com.error.log.1:[Sun Nov 23 13:20:36 2008] [error] [client 69.65.10.202] Options ExecCGI is off in this directory: /home/flexigen/domains/spijtag.com/public_html/scripts/check.cgi spijtag.com.error.log.1:[Sun Nov 23 14:02:04 2008] [error] [client 69.65.10.202] Options ExecCGI is off in this directory: /home/flexigen/domains/spijtag.com/public_html/scripts/check.cgi spijtag.com.log.1:213.169.63.254 - - [23/Nov/2008:08:31:11 -0800] "GET /scripts/check.cgi HTTP/1.0" 403 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" spijtag.com.log.1:69.65.10.202 - - [23/Nov/2008:09:26:38 -0800] "GET /scripts/check.cgi HTTP/1.0" 403 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" spijtag.com.log.1:69.65.10.202 - - [23/Nov/2008:10:14:08 -0800] "GET /scripts/check.cgi HTTP/1.0" 403 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" spijtag.com.log.1:69.65.10.202 - - [23/Nov/2008:12:04:32 -0800] "GET /scripts/check.cgi HTTP/1.0" 403 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" spijtag.com.log.1:69.65.10.202 - - [23/Nov/2008:13:20:36 -0800] "GET /scripts/check.cgi HTTP/1.0" 403 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" spijtag.com.log.1:69.65.10.202 - - [23/Nov/2008:14:02:04 -0800] "GET /scripts/check.cgi HTTP/1.0" 403 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" # </snip> From the above the errors from the errors from the error log, the good accesses from the access log. I don't know yet if this was the source of the spam or not. Here's what I did about the spam: I reconfiged our squirrelmail webmail file to use the sendmail link rather than smtp to port 25 to send email, and I blocked all email from port 127.0.0.1. Since then the emails are being blocked by my local blocklist. There have been no mails since I suspended the site, but I'll check again in the morning to see if that's resolved the issue. Again, thanks for your help. Jeff -- Jeff Lasman, Nobaloney Internet Services P.O. Box 52200, Riverside, CA 92517 Our jplists address used on lists is for list email only voice: +1 951 643-5345, or see: "http://www.nobaloney.net/contactus.html";
