On Tue, Jan 27, 2026 at 09:21:59AM +0100, Tom Szilagyi said:
Most humans use browsers with JS; most bots (including all the LLM scrapers) do not bother. So this is transparent for actual humans browsing around. For my case, this approach solved the problem completely (for now).
I have found more and more that AI scrapers tend to try with a simplistic bot first and then come back with what looks like headless Chrome. I've seen lots of them that are fully executing JavaScript just like most modern search engine crawlers. I've taken (for better or worse) to flat out dropping traffic from a number of ASNs at the edge along with everything else.
-- Please direct replies to the list.
