Am 16.07.2025 um 10:48:39 Uhr schrieb Andrew Latham via NANOG: > 1. What tools or setups have people found most successful for dealing > with bots/scrapers that do not respect robots.txt for example?
Place a link to a file that is hidden to normal people. Exclude the directory via robots.txt. Then use fail2ban to block all IP addresses that poll the file. -- Gruß Marco Send unsolicited bulk mail to [email protected]
pgp2XlNDxp6eS.pgp
Description: OpenPGP digital signature
_______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/[email protected]/message/WX65VXNA5F2CMOZLFCXGNPVLMNIY5QV5/
