On 7/1/25 8:22 PM, Constantine A. Murenin via NANOG wrote:
But the bots are not a problem if you're doing proper caching and throttling.
Not all site traffic is cacheable or can be farmed out to a CDN.
Dynamic (especially per-session) requests (think ecommerce) can't be cached.
Putting an item into the shopping cart is typically one of the more
resource driven events.
We have seen bots that will select the buy button and put items into the
cart, possibly to see
any discounts given. You end up with hundreds of active 'junk' cart
sessions on a small site
that was not designed for that much traffic.
Forcing the bot (or a legit customer) to create yet another login to
create a cart can help
but that generates push back from the store owner. The owners don't want
that until
the payment details phase or they want purchasers to be able to do a
guest checkout.
They will point that on amazon.com you don't have to login to put an
item in the cart.
Rate limiting is not effective when they come from different ip ranges.
The old days of using
a Class C (/24) as a rate limit key are no longer effective. The bots
come from all over the providers space
(often Azure) but can be from any of the larger providers and often from
different regions.
if you throttle EVERYONE then legit customers can get locked out with
429 or even 503s
And has been pointed out. Relying on the browser string is no longer
effective. They use
common strings and change them dynamically.
Sincerely,
William Kern
PixelGate Networks.
_______________________________________________
NANOG mailing list
https://lists.nanog.org/archives/list/[email protected]/message/3GYV3XRRRHNGRPCOXLYZR4SGYNNHKRWY/
