Hi Aleks,
On Fri, Nov 06, 2026 at 11:36:28AM +0100, Aleksandar Lazic wrote:
> Hi.
>
> While testing a latency aware variant of "balance random" (an RFC will
> follow), I noticed that the comparison of the draws is skipped more
> often than expected. random(<draws>) draws the servers on the
> consistent hash ring with replacement, so a draw can return the best
> server found so far. There is nothing to compare then, and the draw is
> lost. With 4 servers and 2 draws, this happens for one choice out of
> four, which then behaves like random(1).
>
> The first patch draws again in this case, at most <draws> extra times
> so that a farm with a single usable server does not loop. It is a few
> lines in get_server_rnd(), plus one sentence in the documentation.
>
> Measured locally with ideal servers (fixed delays, 30000 requests after
> a warm-up), share of the requests sent to the slow servers:
>
> farm random(2) before -> after
> 4 servers, 1 at 50ms 8.73% -> 5.16%
> 8 servers, 2 at 50ms 7.5% -> 6.80%
Interesting. However I'm not fond of looping on extra draws, and we
don't need to because the consistent hash function already takes as
an argument the server to avoid, so we can use it to pass the previous
preferred server, which gives an even better distribution because
instead of having 25% chance of hitting the same server on each draw,
now we're down to 0% so we don't need to increase the count at all.
I'm testing with this config, having 1 server at 50ms and N-1 servers
at 5ms:
defaults
mode http
timeout client 5s
timeout server 5s
timeout connect 5s
frontend stats
bind :8181
stats uri /
listen f
bind :8001
balance random(2)
server s0 127.0.0.1:8003 # slow
server s1 127.0.0.1:8002
server s2 127.0.0.1:8002
server s3 127.0.0.1:8002
#server s4 127.0.0.1:8002
#server s5 127.0.0.1:8002
#server s6 127.0.0.1:8002
#server s7 127.0.0.1:8002
frontend fast
bind :8002
http-request pause 5ms
http-request return status 200
frontend slow
bind :8003
http-request pause 50ms
http-request return status 200
Only the first server is slow (for ease of testing). In master I'm
getting this distribution for 100k req, 100 concurrent connections,
on 4 servers (server, max concurrent, total):
srv max total
s0 63 8525
s1 27 28296
s2 28 31049
s3 29 32130
Request rate: 11.1k
For 8 servers:
srv max total
s0 29 2248
s1 17 13874
s2 17 13869
s3 18 13996
s4 18 14316
s5 17 14325
s6 16 13758
s7 16 13614
Request rate: 15.8k
Now with this patch applied:
--- a/src/backend.c
+++ b/src/backend.c
@@ -584,7 +584,7 @@ struct server *get_server_rnd(struct stream *s, const
struct server *avoid)
do {
prev = curr;
hash = statistical_prng();
- curr = chash_get_server_hash(px, hash, avoid);
+ curr = chash_get_server_hash(px, hash, avoid ? avoid : prev);
if (!curr)
break;
4 servers:
srv max total
s0 29 3572
s1 28 31380
s2 29 32513
s3 29 32535
Request rate: 14.6k (+33%)
8 servers:
srv max total
s0 17 1669
s1 15 14007
s2 16 14010
s3 15 14059
s4 16 14205
s5 16 14215
s6 16 13968
s7 15 13867
Request rate: 16.6k (+5%)
The gains are clear, the ratios follow the response times, and the
distribution is also much smoother. I remember that when I wrote
that old article about P2C I mentioned that more draws were cleaner
(3+ is better) and allowed to avoid such cases as ending on the same
server, but by then that didn't flow into the code though.
> The second patch documents the other source of unevenness I found:
> with the default weight of 1, each server only gets 16 points on the
> ring, so the shares of the servers vary between about 2% and 4% with
> 32 servers. The same higher weight for all servers, such as 100, makes
> them almost even. I only documented it, as changing the number of
> points would change the distribution of the existing setups.
>
> I would be glad about any feedback, also if you would prefer to keep
> the current behavior of random for some reason I did not think of.
I'm going to merge my smaller change, crediting you for the suggestion,
and your doc patch.
Thank you!
Willy