Jesus Cea writes: > Since AI scrapers are not obeing "robots.txt" and they pretend to be a > regular browsers, they are coming from thousands of IPs at a time and > they do costly mercurial requests hammering my servers... I wonder > what to do about it.
I've struggled with this for a month, I _think_/hope I've finally nailed it using fail2ban with ipsets, but my configuration is still a bit messy and not fully automated. If anyone has a clean working configuration for that or any other simple nftables-based approach, please post. This way you actually stop the bad-guys at the front door, rather than waiting until they get to apache/nginx/whatever server, so consume much less load. When I get mine finally sorted I'll share. ht -- Henry S. Thompson _______________________________________________ Mercurial mailing list Mercurial@lists.mercurial-scm.org https://lists.mercurial-scm.org/mailman/listinfo/mercurial
