On 16/3/2026 18:02, Fred Wright wrote:
On Mon, 16 Mar 2026, Joshua Root wrote:
On 16/3/2026 09:32, Fred Wright wrote:
This approach to bot-blocking is pretty lame. There's nothing to
stop the bots from using User-Agent strings from current browsers.
What then, require everyone to use lynx? IP-based filtering would be
more robust and less user-unfriendly.
Blocking the IP ranges that the scraping comes from would block most
of the internet, and almost certainly block more legitimate users than
we do now. UA blocking is not a great solution but it works (by which
I mean, the site is usable, which it would not be otherwise). The only
real alternative is to deploy something like Anubis, which would also
block legitimate robots.txt-respecting scrapers.
Not *ranges*, the specific IP addresses causing trouble. It could even
be fully automated based on rate limiting.
That's even less effective. Individual IP addresses are only making a
handful of requests each. These systems are specifically designed to
thwart blocking.
- Josh
