> > Most of the tar pits I’ve seen want to poison the AI training data well, > but we’re really mostly interested in not having our servers hammered. > Giving the bots more content seems counter-intuitive to that. >
>From what I've read about Nepenthes and Iocaine, they aren't just simply about poisoning the AI training data. The area where they trap the crawlers is designed to be a low bandwidth area (both in terms of speed of requests and responses, as well as CPU resources), so that it forces the crawlers to scrape very slowly... hence the name "tar pit". But, since I've never actually implemented either of these myself, who knows whether that's actually true... <shrug> -- Jason Liu On Mon, Mar 16, 2026 at 6:15 PM Clemens Lang <[email protected]> wrote: > Hi Jason, > > > On 16. Mar 2026, at 19:34, Jason Liu <[email protected]> wrote: > > > > If the concern is bots/scrapers/crawlers (which are increasingly > AI-powered these days), then have you considered "tar pit" software, such > as Nepenthes or Iocaine? > > > > > https://arstechnica.com/tech-policy/2025/01/ai-haters-build-tarpits-to-trap-and-trick-ai-scrapers-that-ignore-robots-txt/ > > > > CloudFlare also now offers a product to trap misbehaving crawlers, > called AI Labyrinth: > > > > https://blog.cloudflare.com/ai-labyrinth/ > > I wanted to avoid deploying yet another software package, but if we end up > going down that route, I’d probably start with Anubis. Most of the tar pits > I’ve seen want to poison the AI training data well, but we’re really mostly > interested in not having our servers hammered. Giving the bots more content > seems counter-intuitive to that. > > As for CloudFlare, I’m not a fan of relying on it, but at this point I’m > open to anything. IIRC we have an existing relationship with a CDN > provider, maybe we should move Trac behind that. They’re not Cloudflare, > but I guess they might have a similar offering. > > Ryan, Joshua, would you happen to know whether our CDN has such > functionality? > > > -- > Clemens
