> I don’t care why, I’m not changing my browser configuration to > accomodate whatever it is they think they need to make it work, but > it does mean that more and more of the web has become inaccessible > for me.
As a guy who has to keep a variety of web sites running, I can explain the why: 1. Spiders and robots, many driven by the current AI craze, have increased in number, mendicacity, lawlessness, and aggressiveness by orders of magnitude. If you're running a web site that provides content via some sort of dynamic mechanism (content management system) that runs code and does database queries to generate each response page, the new level of offered load squashes your server into a greasy spot. 2. These spiders (a) lie about who they are, pretending to be real humans; (b) don't obey robots.txt; (c) retrieve the same page multiple times at the same instant, sometimes using different user agent strings, to try to be sure they get the page in the face of increasing server hostility to their bullshit. 3. These bots don't care about request rate; they just expect you to deal with it. Will the VPS where you host your ccmp Wordpress install keep up with dozens of page requests per second? Because if the slop machine gets interested in something, those requests will come, and will probably flatten you. 4. Search and (especially) AI companies have had to resort to making end user machines fetch the content they want. Smart TVs, phones, tablets, and other devices are increasingly being used. These devices live in IP address space that's assigned to DSL, cable modems, cell companies, etc. ... and the difficulty of surviving the onslaught: 1. Because these requests come from where real users live, blocking by IP address blocks the real users. 2. Because the user agent strings are fabricated to be indistinguishable from real user browsers, blocking by them blocks the real users. 3. Blocking by behavior has some promise, but you really need insight into behavior across a wide swath of the internet, unless you're that big yourself. I.e. Cloudflare and Google and Amazon have a chance. Hobbyists don't. For a dynamic web site, that basically leaves putting some sort of challenge in front of your site that humans can solve easily, but the bots can't (or don't want to bother), coupled with broader intelligence on trends across the 'net. That looks, unsurprisingly, like Cloudflare and similar services. Unfortunately, those kinds of challenges, and the interface between them and the protected website, is largely dependent on javascript and cookies. Grumble. Welcome to late stage capitalism. The future, for those of us who hate the technologies, budgets, and ick factor required to survive the post-web sewer, probably lies in static web sites. Think tools like Hugo, Jekyll, Pelican. Edit the page on your local machine. Run the generator. Synchronize the updated flat HTML pages to the server. Web servers deliver flat files with remarkable efficiency. sendfile(2) is damn fast. Additional upside: it's hard to hack a web app that has no code running on the server. De
