On Jan 27, 2006, at 10:41, Larry Cook wrote:
How do you keep out the bad ones, the ones that ignore robots.txt?
The bad ones usually _read_ robots.txt to figure out where the "juicy
stuff" is.
So you can do:
Disallow: /robottrap.html
And then have something tail your access log and instantly iptables
anything that accesses /robottrap.html.
-Bill
-----
Bill McGonigle, Owner Work: 603.448.4440
BFC Computing, LLC Home: 603.448.1668
[EMAIL PROTECTED] Cell: 603.252.2606
http://www.bfccomputing.com/ Page: 603.442.1833
Blog: http://blog.bfccomputing.com/
VCard: http://bfccomputing.com/vcard/bill.vcf
_______________________________________________
gnhlug-discuss mailing list
[email protected]
http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss
