i wrote a program that counts requests by IP and frequency. That and some fancy code to teduce server load and the bots are blocked fairly well. I capture the attempts and lot them so I can see where and how theyre doing it. Inhave noticed many bots use the same exact probe progression, there must be a popular scraper bot you can download b
On Tue, Jul 7, 2026 at 6:58 PM Dennis Boone via cctalk < [email protected]> wrote: > > I don’t care why, I’m not changing my browser configuration to > > accomodate whatever it is they think they need to make it work, but > > it does mean that more and more of the web has become inaccessible > > for me. > > As a guy who has to keep a variety of web sites running, I can explain > the why: > > 1. Spiders and robots, many driven by the current AI craze, have > increased in number, mendicacity, lawlessness, and aggressiveness by > orders of magnitude. If you're running a web site that provides content > via some sort of dynamic mechanism (content management system) that runs > code and does database queries to generate each response page, the new > level of offered load squashes your server into a greasy spot. > > 2. These spiders (a) lie about who they are, pretending to be real > humans; (b) don't obey robots.txt; (c) retrieve the same page multiple > times at the same instant, sometimes using different user agent strings, > to try to be sure they get the page in the face of increasing server > hostility to their bullshit. > > 3. These bots don't care about request rate; they just expect you to > deal with it. Will the VPS where you host your ccmp Wordpress install > keep up with dozens of page requests per second? Because if the slop > machine gets interested in something, those requests will come, and will > probably flatten you. > > 4. Search and (especially) AI companies have had to resort to making > end user machines fetch the content they want. Smart TVs, phones, > tablets, and other devices are increasingly being used. These devices > live in IP address space that's assigned to DSL, cable modems, cell > companies, etc. > > ... and the difficulty of surviving the onslaught: > > 1. Because these requests come from where real users live, blocking by > IP address blocks the real users. > > 2. Because the user agent strings are fabricated to be indistinguishable > from real user browsers, blocking by them blocks the real users. > > 3. Blocking by behavior has some promise, but you really need insight > into behavior across a wide swath of the internet, unless you're that > big yourself. I.e. Cloudflare and Google and Amazon have a chance. > Hobbyists don't. > > For a dynamic web site, that basically leaves putting some sort of > challenge in front of your site that humans can solve easily, but the > bots can't (or don't want to bother), coupled with broader intelligence > on trends across the 'net. That looks, unsurprisingly, like Cloudflare > and similar services. > > Unfortunately, those kinds of challenges, and the interface between them > and the protected website, is largely dependent on javascript and > cookies. Grumble. > > Welcome to late stage capitalism. > > The future, for those of us who hate the technologies, budgets, and ick > factor required to survive the post-web sewer, probably lies in static > web sites. Think tools like Hugo, Jekyll, Pelican. Edit the page on > your local machine. Run the generator. Synchronize the updated flat > HTML pages to the server. > > Web servers deliver flat files with remarkable efficiency. sendfile(2) > is damn fast. > > Additional upside: it's hard to hack a web app that has no code running > on the server. > > De >
