On 2025-10-17 15:25, Max Chernoff via ntg-context wrote: > Hi Taco, > > On Fri, 2025-10-17 at 14:59 +0200, Taco Hoekwater wrote: >> I agree with the sentiment, but the problem I see is that the Internal >> Error messages are in fact mostly caused by web crawlers overloading >> the system. > > The scraper bots also took down the SVN interface on tug.org a few > months ago, and adding Anubis fixed things. Norbert also had similar > problems on texlive.info, and Anubis fixed things there too. Without > Anubis, we were getting 1MM requests per day, but with it enabled, it > was down to 2k requests per day.
Hi Taco and Max, both https://gitlab.gnome.org/ and https://gitlab.freedesktop.org/ had also to implement it. At least an Invidious instance (https://youtubenuker.lol) uses go-away (https://git.gammaspectra.live/git/go-away). Just in case it might help, Pablo ___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki! maillist : [email protected] / https://mailman.ntg.nl/mailman3/lists/ntg-context.ntg.nl webpage : https://www.pragma-ade.nl / https://context.aanhet.net (mirror) archive : https://github.com/contextgarden/context wiki : https://wiki.contextgarden.net ___________________________________________________________________________________
