Once upon a time, Marco Moock <[email protected]> said: > Place a link to a file that is hidden to normal people. Exclude the > directory via robots.txt. > > Then use fail2ban to block all IP addresses that poll the file.
The problem with a lot of the "AI" scrapers is that they're apparently using botnets and will often only make a single request from a given IP address, so reactive blocking doesn't work (and can cause other issues, like trying to block 100,000 IPs, which fail2ban for example doesn't really handle well). -- Chris Adams <[email protected]> _______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/[email protected]/message/AFJF4UQJZW6ALTY6SA7OHBN2AZC72SZQ/
