Robert

This is a good observation. It has been a decade or two since I worked
in the hosting world. Back in the days of hosting work I think I did
something like a an IPTABLES --recent on high number of new
connections with a 5min block.


I have made several observations and developed a few ideas. This whole
process is like the telemarketer torture systems we discussed back in
the Asterisk project.

Some of my thinking is around

A. What is a search bot verses an AI bot verses a vulnerability
scanner verses an email address scraper (I know but documenting it has
shown the filter issues)
B. What is the CPU, Logging resource usage and were is the balance on inspection
C. Are there dynamic lists like SpamHaus DROP of AI scrapers?
D. For or against AI scraper bots?
E. Response Rate Limiting options verses drop or reject
F. Scan depth issues. (Gitea instance and per commit diff getting scanned)

On Fri, Jul 18, 2025 at 4:13 PM Robert L Mathews via NANOG
<nanog@lists.nanog.org> wrote:
>
> On Jul 16, 2025, at 9:48 AM, Andrew Latham via NANOG <nanog@lists.nanog.org> 
> wrote:
> >
> > 2. What tools for response rate limiting deal with bots/scrapers that
> > cycle over a large variety of IPs with the exact same user agent?
>
>
> If the bots are impersonating real browser User-Agents, and you use something 
> like ModSecurity that can examine HTTP headers, you can look at a few 
> requests and probably find that they send or omit things compared to real 
> browsers.
>
> Today, for example, I blocked some of the requests from a botnet that often 
> sends this pair of headers:
>
>  User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 
> (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36
>  Sec-Ch-Ua-Platform: "macOS"
>
> Note the mismatch of "Windows NT" vs. "macOS": it appears the bot randomizes 
> "Sec-Ch-Ua-Platform" but not the "User-Agent", so a good percentage of their 
> requests show this mismatch.
>
> Another recent high volume botnet impersonating Chrome/134 is sending this 
> header:
>
>  Referrer: https://www.google.com/
>
> [sic]: They forgot to misspell "Referer".
>
> Most botnets I look at have multiple "tells" like this in the HTTP headers. 
> You have to be mindful to avoid false positives from proxies that mess with 
> headers, but it's otherwise an effective way to block them and stop them from 
> consuming CPU time.
>
> Whether this is worth your time is a different matter. It's worth mine 
> because we host thousands of sites, but I probably wouldn't waste the effort 
> on it if it was just my own site, unless the botnet was making the site not 
> work.
>
> --
> Robert L Mathews
>
> _______________________________________________
> NANOG mailing list
> https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/DXWVYDR47HOLJBQQNZQKAK242ASV2SWY/



-- 
- Andrew "lathama" Latham -
_______________________________________________
NANOG mailing list 
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/OGC5PZ7U763BFPECWUUIBQ6XFF2AD2DQ/

Reply via email to