As far as I know, there are no accessibility issues with Anubis's proof-of-work scheme. Assistive technologies sit on top of standard browsers, not parallel to them. They're not headless, and they don't have their own user agents.
Stephanie Leary Front End Developer Equinox Open Library Initiative stephanie.le...@equinoxoli.org https://www.equinoxOLI.org phone: 877-OPEN-ILS (673-6457) On Thu, Jul 10, 2025 at 11:27 AM John Merriam via Evergreen-dev < evergreen-dev@list.evergreen-ils.org> wrote: > Hi Mike. > > I hear you on the old Chrome user agents. For now that is what we're > going with since it works. So far no complaints. > > Thank you very much for the pointer to LP2113979 We will look into that. > > Regarding Anubis, I'm not a fan at the moment unless there is no other > way. I could be wrong but I think it will break accessibility (would it > work with software blind patrons use)? I also have a problem with > wasting all that energy. It's bad enough that AI data centers are on > track to destroy the planet. I don't want to have to waste tons of > energy fighting them. > > > On 7/10/2025 12:07 PM, Mike Rylander wrote: > > Some things to consider, inline below... > > > > On Thu, Jul 10, 2025 at 11:25 AM John Merriam via Evergreen-dev > > <evergreen-dev@list.evergreen-ils.org> wrote: > >> Hello. > >> > >> This will block Chrome older than 110 (over 2 years old) in Nginx: > >> > >> if ($http_user_agent ~* > "(Chrome/10[0-9]\.|Chrome/[0-9][0-9]\.|Chrome/[0-9]\.)") { > >> return 403; > >> } > >> > >> which put a stop to it for now for us. > >> > > Please be careful. In addition to patrons with old browsers (there > > are plenty out there, unfortunately) there are some black-box kiosks > > out in the wild that are used for selfcheck and in-building OPAC > > machines which use an older Chrome (and are not free to upgrade). > > > >> Changing user agents is trivial though so finding other blockable > patterns such as in URLs would be good. I didn't find a good pattern to > the URLs yet but I was only able to look at that quickly. I plan on > circling back around to that at some point. > >> > >> I don't think blocking by IP will work against what seems to be a > distributed AI botnet. A few months ago we had our data center partners > block all non-US IPs. That worked for a few months but even that doesn't > work anymore. We see AI bot traffic coming from US residential IP ranges. > A gigantic question I have is how are they appearing to come from > residential IPs and how could that be stopped? > >> > >> We plan to profile Evergreen looking for slow code that could maybe be > improved but that will be a big project. > >> > > I invite more eyes, of course, but "big project" is a bit of an > > understatement. ;) > > > > Please be careful when testing something that seems "slow" in > > isolation -- making code X 10% faster will often make > > seemingly-unrelated code Y 90% slower. > > > >> We also plan to hook a WAF with machine learning into Nginx and see > what that can do. Another big project. > >> > >> We may also put captcha on more parts of the OPAC. We have someone > working on that. > >> > > Have you looked at https://bugs.launchpad.net/evergreen/+bug/2113979? > > With some refinement of the URL space where the not-a-bot cookie is > > required, this is shaping up to be a good first-order bot killer. > > > >> I can allocate more resources to the OPAC but that seems like letting > them win and they will probably eventually exhaust that as well. > >> > >> Anubis is a nuclear option I would like to avoid. > >> > > I'm curious why you see this as a nuclear option. Granted, most AI > > scrapers right now (at least, AFAICT) seem to be essentially > > stateless, so it may be overkill compared to the LP bug linked above, > > but it's fairly straight-forward to set up and maintain. The only > > drawback right now is that you have to use just one instance, which > > could become a bottleneck in a very "wide" EG setup. > > > >> Also don't want to turn to something like Cloudflare. > >> > > It's certainly not cost effective for the Library space... > > > >> Please do share any findings and I will as well. > >> > >> Thanks > >> > >> > >> On 7/10/2025 10:53 AM, Josh Stompro via Evergreen-dev wrote: > >> > >> One piece of this puzzle that I would like to understand better is how > the bad actors are targeting our sites with thousands to hundreds of > thousands of unique IP endpoints each day. And I just saw this article > come out about how 1 million browsers have installed extensions that turn > the users browser into scrapers. > >> > >> > https://arstechnica.com/security/2025/07/browser-extensions-turn-nearly-1-million-browsers-into-website-scraping-bots/ > >> > >> Josh > >> > >> > >> On Thu, Feb 13, 2025 at 3:49 PM Shula Link via Evergreen-dev < > evergreen-dev@list.evergreen-ils.org> wrote: > >>> It's not just Evergreen sites. I had to block all traffic from Hong > Kong to our system website after we had a greater than 10x increase in > visitors overnight. I tried doing it by IP, but they just changed, so it > ended up just being easier to just block everything. > >>> > >>> Shula Link (she/her) > >>> Systems Services Librarian > >>> Greater Clarks Hill Regional Library > >>> sl...@columbiacountyga.gov | sl...@gchrl.org > >>> 706-447-6702 > >>> > >>> > >>> On Thu, Feb 13, 2025 at 4:46 PM Blake Graham-Henderson via > Evergreen-dev <evergreen-dev@list.evergreen-ils.org> wrote: > >>>> All, > >>>> > >>>> I almost replied with the arstechnica article that Josh linked when > the thread was started. But I decided not to put it out there until I had > setup a test system to see if I could get that code working. A tarpit, I > think, serves them right. And, of course, the whole issue is destined to > receive the fate of spam and spam filters forever and ever. > >>>> > >>>> It was a serendipitous timed article. It's existence at this moment > in time signals to me that this isn't a "just us" problem. It's the entire > planet. > >>>> > >>>> -Blake- > >>>> Conducting Magic > >>>> Will consume any data format > >>>> MOBIUS > >>>> > >>>> On 2/13/2025 3:10 PM, Josh Stompro via Evergreen-dev wrote: > >>>> > >>>> Jeff, thanks for bringing this up on the list. > >>>> > >>>> We are seeing a lot of requests like > >>>> "GET /eg/opac/mylist/delete?anchor=record_184821&record=184821" > from never seen before IPs, and they make 1-12 requests and then stop. > >>>> > >>>> And they seem like they usually have a random out of date chrome > version in the user agent string. > >>>> Chrome/88.0.4324.192 > >>>> Chrome/86.0.4240.75 > >>>> > >>>> I've been trying to slow down the bots by collecting logs and > grabbing all the obvious patterns and blocking netblocks for non US > ranges. ipinfo.io offers a free country & ASN database download that > I've been using to look up the ranges and countries. ( > https://ipinfo.io/products/free-ip-database) I would be happy to share a > link to our current blocklist that has 10K non US ranges. > >>>> > >>>> I've also been reporting the non US bot activity to > https://www.abuseipdb.com/ just to bring some visibility to these bad > bots. I noticed initially that many of the IPs that we were getting hit > from didn't seem to be listed on any blocklists already, so I figured some > reporting might help. I'm kind of curious if Evergreen sites are getting > hit from the same IPs, so an evergreen specific blocklist would be useful. > If you look up your bot IPs on abuseipdb.com you can see if I've already > reported any of them. > >>>> > >>>> I've also been making use of block lists from > https://iplists.firehol.org/ > >>>> Such as > >>>> https://iplists.firehol.org/files/cleantalk_30d.ipset > >>>> https://iplists.firehol.org/files/botscout_7d.ipset > >>>> https://iplists.firehol.org/files/firehol_abusers_1d.netset > >>>> > >>>> We are using HAProxy so I did some looking into the CrowdSec HAProxy > Bouncer (https://docs.crowdsec.net/u/bouncers/haproxy/) but I'm not sure > that would help since these IPs don't seem to be on blocklists. But I may > just not quite understand how CrowdSec is supposed to work. > >>>> > >>>> HAProxy Enterprise has a ReCaptcha module that I think would allow us > to feed any non-us connections that haven't connected before through a > recaptcha, but the price for HAProxy Enterprise is out of our budget. > https://www.haproxy.com/blog/announcing-haproxy-enterprise-3-0#new-captcha-and-saml-modules > >>>> > >>>> There is also a fairly up to date project for adding Captchas through > haproxy at > >>>> https://github.com/ndbiaw/haproxy-protection, This looks promising > as a transparent method, requires new connections to perform a javascript > proof of work calculation before allowing access. Could be a good > transparent way of handling it. > >>>> > >>>> We were taken out by ChatGTP bots back in December, which were a bit > easier to block the netblocks since they were not as spread out. I > recently saw this article about how some people are fighting back against > bots that ignore robots.txt, > https://arstechnica.com/tech-policy/2025/01/ai-haters-build-tarpits-to-trap-and-trick-ai-scrapers-that-ignore-robots-txt/ > >>>> > >>>> Josh > >>>> > >>>> On Mon, Jan 27, 2025 at 6:33 PM Jeff Davis via Evergreen-dev < > evergreen-dev@list.evergreen-ils.org> wrote: > >>>>> Hi folks, > >>>>> > >>>>> Our Evergreen environment has been experiencing a higher-than-usual > volume of unwanted bot traffic in recent months. Much of this traffic looks > like webcrawlers hitting Evergreen-specific URLs from an enormous number of > different IP addresses. Judging from discussion in IRC last week, it sounds > like other EG admins have been seeing the same thing. Does anyone have any > recommendations for managing this traffic and mitigating its impact? > >>>>> > >>>>> Some solutions that have been suggested/implemented so far: > >>>>> - Geoblocking entire countries. > >>>>> - Using Cloudflare's proxy service. There's some trickiness in > getting this to work with Evergreen. > >>>>> - Putting certain OPAC pages behind a captcha. > >>>>> - Deploying publicly-available blocklists of "bad bot" > IPs/useragents/etc. (good but limited, and not EG-specific). > >>>>> - Teaching EG to identify and deal with bot traffic itself (but > arguably this should happen before the traffic hits Evergreen). > >>>>> > >>>>> My organization is currently evaluating CrowdSec as another possible > solution. Any opinions on any of these approaches? > >>>>> -- > >>>>> Jeff Davis > >>>>> BC Libraries Cooperative > >>>>> _______________________________________________ > >>>>> Evergreen-dev mailing list > >>>>> Evergreen-dev@list.evergreen-ils.org > >>>>> http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-dev > >>>> > >>>> _______________________________________________ > >>>> Evergreen-dev mailing list > >>>> Evergreen-dev@list.evergreen-ils.org > >>>> http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-dev > >>>> > >>>> > >>>> _______________________________________________ > >>>> Evergreen-dev mailing list > >>>> Evergreen-dev@list.evergreen-ils.org > >>>> http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-dev > >>> _______________________________________________ > >>> Evergreen-dev mailing list > >>> Evergreen-dev@list.evergreen-ils.org > >>> http://list.evergreen-ils.org/cgi-bin/mailman/listinfo/evergreen-dev > >> > >> _______________________________________________ > >> Evergreen-dev mailing list -- evergreen-dev@list.evergreen-ils.org > >> To unsubscribe send an email to > evergreen-dev-le...@list.evergreen-ils.org > >> > >> -- > >> John Merriam > >> Director of Information Technology > >> Bibliomation, Inc. > >> 24 Wooster Ave. > >> Waterbury, CT 06708 > >> 203-577-4070 > >> > >> _______________________________________________ > >> Evergreen-dev mailing list -- evergreen-dev@list.evergreen-ils.org > >> To unsubscribe send an email to > evergreen-dev-le...@list.evergreen-ils.org > > -- > John Merriam > Director of Information Technology > Bibliomation, Inc. > 24 Wooster Ave. > Waterbury, CT 06708 > 203-577-4070 > > _______________________________________________ > Evergreen-dev mailing list -- evergreen-dev@list.evergreen-ils.org > To unsubscribe send an email to evergreen-dev-le...@list.evergreen-ils.org >
_______________________________________________ Evergreen-dev mailing list -- evergreen-dev@list.evergreen-ils.org To unsubscribe send an email to evergreen-dev-le...@list.evergreen-ils.org
