On 2026/10/31 19:12:28 Walter Underwood wrote:
> Deep paging bots were my nemesis. I had to limit the page depth in the front
> end twice, once at Netflix (Solr 1.x) and once at Chegg. Ask for page 51, you
> get page 50.
>
> Had to fix the first one from a hotel room in Park City. Netflix gave us
> partial funding to go to the Sundance Film Festival. It still wasn’t cheap,
> but it was nice. Of course, the site was busted as soon as we checked in.
>
> There always seems to be some new stupid bot behavior that we have to defend
> against.
>
> wunder
> Walter Underwood
> [email protected]
> http://observer.wunderwood.org/ (my blog)
>
> > On Oct 31, 2026, at 11:56 AM, Dmitri Maziuk <[email protected]> wrote:
> >
> > On 10/31/26 12:29, Gus Heck wrote:
> >> Every situation is different, and that design is what Solr used to
> >> recommend before we added the rule based auth plugin. It all depends on
> >> how critical internal security is vs external security. Very large
> >> companies sometimes have more risk from internal threats, small companies
> >> can often afford to trust internal folks more. The intent of the auth
> >> plugins we have is to allow for internal security, though I think they
> >> deviate from expected norms and are more complex than they need to be.
> > Sadly, with the AI bubble of 2026 merging with the cybersecurity bubble of
> > 2023, it is becoming less and less about the actual security and more about
> > the theater. And it's only going to get worse.
> >
> > In our case about the worst thing 3viL h4x0rZ can do by breaking into Solr,
> > specifically, is to feed visitors false information that is part of
> > historical record and can be independently verified from any number of
> > other sources... i.e. not different from said visitors asking ChatGPT. So
> > the main threats in our environment is internal tenable scans. And crawler
> > bots that every once in a while find a way to get past the filters and DDoS
> > the cluster.
> >
> > (The latest batch, BTW, was searching for `xxx` in a DateRange field and
> > had enough requests/minute that `SolrExceptoin: Couldn't parse date
> > becasue: Improperly formatted datetime:xxx` exceptions first pegged the CPU
> > at 110% which -- presumably -- starved the garbage collector, and then the
> > entire JVM OOM'ed after a while. That's the actual real-world threat, not
> > the CVE rubbish that take the entire Calude months to find and tens of
> > hours to actually exploit.)
> >
> > Dima
> >
>
>
If you thought deep paging bots was a challenge, wait till when we flip the
script and give agents our schema, query options, and here are my rerankers,
now you build your own custom query! You may be interested in this dev
thread: https://lists.apache.org/thread.html/2lvrz5qozkwbxv5j52jon66zms4hbwpp
