Xavier Mertens wrote:
I've a problem with an Apache web server hit by f*cking spammers...
I would like to filter some URLs (unused but still used by the bots) *BEFORE*
they reach the httpd processes. What could be the best method? pf? something
else?
PF doesn't look at URL content, so can't be use to filter that type of
stuff.
What I do use extensively however that does a great job is the rewrite
module for virus attack and the like and I also put two files for the
bot that crawler sites with the proper directory not to scan and all
that do not respect that get block in PF after they access that file as
I log that in sql and pass the data to all servers.
I also put URL in the copyright section of my sites to point to the
forbidden directory of the bots and as such each low life that import my
site without rights, get also block via PF!
I do catch many bad bots that do not act properly and it's nice to see.
So, it's a combination of things.
Here is some extract for example of httpd.conf for the rewrite module.
Then you add and removed them as you see fit and new one coming up.
You can also limit the number of connections from the same IP address
with pf as well for very abusive crawlers, or bad DDoS guys, etc.
Hope this give you some ideas anyway.
Best,
Daniel
### Special section for stopping bad traffic and DDoS attack type, etc
# This list are all global rewrite rules that can be turn on per virtual
server.
RewriteCond %{HTTP_REFERER} ^$
RewriteCond %{HTTP_USER_AGENT} ^$
RewriteCond %{REQUEST_URI} ^/$
RewriteRule ^/.* http://%{REMOTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)cmd.exe(.*)$ http://%{REMOTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)root.exe(.*)$ http://%{REMOTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)shell.exe(.*)$ http://%{REMOTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)\/_vti_bin\/(.*)$ http://%{REMOTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)\/_vti_cnf\/(.*)$ http://%{REMOTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)\/_vti_inf\/(.*)$ http://%{REMOTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)\/_vti_inf.html http://%{REMOTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)\/scripts\/\.\.(.*)$ http://%{REMOTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)\/_mem_bin\/(.*)$ http://%{REMOTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)\/msadc\/(.*)$ http://%{REMOTE_ADDR}/
[NC,L,E=nolog:1]
RewriteRule (.*)\/MSOffice/cltreq.asp(.*)$ http://%{REMOTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)\/c\/winnt\/(.*)$ http://%{REMOTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)\/d\/winnt\/(.*)$ http://%{REMOTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)\/x80\/(.*)$ http://%{REMOTE_ADDR}/
[L,E=nolog:1]
RewriteRule (.*)\/x90\/(.*)$ http://%{REMOTE_ADDR}/
[L,E=nolog:1]
