You mentioned 250k requests/day, but you did not characterise the population spread.This is also "whack-a-mole" -- you asked for "population spread" (my comfort level in politics is low), but the spread is somewhat close to world population - led by China, Pakistan, Vietnam, Brazil and Microsoft. Conspicuously absent are Russia and Google. This is pure math in my microcosm.
Maybe did I use the wrong word, but by "spread" I meant to talk about the diversity of IP addresses: are they always different or are the same coming back over and over?
In the former case, you are most probably hitting the LRU eviction on your zone, hence virtually "resetting" their rate-limit, and allow for more requests to pass than you would wish. You could try and play with the zone memory size to see if that has an effect or not.
If that is not enough, are there recognisable patterns, such like relatively narrow CIDR ranges they would belong to (/10 or /11 are way too big), which could be linked to recurring organisations?
It's all guess-work after all. At the moment, you are directly working in $binary_remote_address.If you can regroup IP addresses in CIDR ranges, you could apply different rate limits. To do that, you could stack the geo directive feeding a map one, in turn feeding limit_req_zone in the end.
Finally, on top of limiting requests, you could be also limiting connections of the worst offenders with limit_conn. The effectiveness of that added layer will essentially depend on whether those requests are reusing connections or not.
-- Bernard Rosset https://rosset.net/
smime.p7s
Description: S/MIME Cryptographic Signature
