On 1 mei 2013, at 13:31, Graham Leggett <minf...@sharp.fm> wrote: > > The evidence was just explained - a bot that does not get an answer quick > enough gives up and looks elsewhere. > The key words are "looks elsewhere".
For what it is worth - I've been experimenting with this (up till about 6 months ago) on a machine of mine. Having the 200, 403, 404, 500 etc determined by an entirely unscientific 'modulo' of the IP address. Both on the main URL as well as on a few PHP/plesk hole URLs. And have ignored/behaved normal for any source IP that has (ever) fetched robot.txt from the same IP masked by the first 20 bits. That showed that bot's indeed slowdown/do-not-come back so soon if you give them a 403 or similar - but I saw no differences as to which non 200 you give them (not tried slow reply or no reply). Do note though that I was focusing on naughty non-robot.txt fetching bots. Dw
smime.p7s
Description: S/MIME cryptographic signature
