Hi Mike, You could give this a sho: (notice the moved "): HeadDeny "User-Agent: .*MJ12bot.*"
This should deny any request that has 'MJ12bot' anywhere in the User-Agent header, but I am not sure if it is case sensitive. I based this on the RFC (https://tools.ietf.org/html/rfc7231#section-5.5.3) Could anyone confirm (or deny) if the HeadDeny option accepts full regex? If so, what version? (i.e. Perl) If it is indeed full regex I could make a case insensitive version for you. Regards, Jan 2016-03-01 19:13 GMT+01:00 Mike Slinn <[email protected]>: > robots.txt is only honored by the good guys. For bad guys, robots.txt > provides a list of places that they know you don't want them to look at. > Also, if pound blocks the bad guys, the web server won't have to handle > those requests, which can present a significant load. > > > -- > To unsubscribe send an email with subject unsubscribe to [email protected]. > Please contact [email protected] for questions.
