On Mon, 16 Mar 2026, Joshua Root wrote:
On 16/3/2026 09:32, Fred Wright wrote:
This approach to bot-blocking is pretty lame. There's nothing to stop the
bots from using User-Agent strings from current browsers. What then,
require everyone to use lynx? IP-based filtering would be more robust and
less user-unfriendly.
Blocking the IP ranges that the scraping comes from would block most of the
internet, and almost certainly block more legitimate users than we do now. UA
blocking is not a great solution but it works (by which I mean, the site is
usable, which it would not be otherwise). The only real alternative is to
deploy something like Anubis, which would also block legitimate
robots.txt-respecting scrapers.
Not *ranges*, the specific IP addresses causing trouble. It could even be
fully automated based on rate limiting.
Fred Wright
