On Tuesday, September 16th, 2025 at 17:51, Paul Koning via cctalk <[email protected]> wrote:
> If they are honest they will obey robots.txt and you can use that to stop > them. If it doesn't robots.txt doesn't seem to help. If anything it can hurt in this regard. Analyzing my site logs I noticed at least one scraper botnet pulling /robots.txt on its initial connect and then immediately pulling down everything listed in it. Which is also its downfall if you seed the file with traps like a tarpit or a data compression bomb. The Doctor [412/724/301/703/415/510] WWW: https://drwho.virtadpt.net/ Get thee down. Be thou funky.
