Hello James,
The Balance random should do exactly what you want in HAProxy 1.9 :
Check out
https://www.haproxy.com/blog/haproxy-1-9-has-arrived/#random-load-balancing-algorithm
"We’ve added a new random load-balancing algorithm. When used, a random
number will be chosen as the key for the consistent hashing function. In
this mode, server weights are respected. Dynamic weight changes take
effect immediately, as do new server additions."
So to get the desired effect, you could use balance random without
server weights: this should yield an uniform distribution.
However, the socket approach is probably the best approach if you really
want to have the 503-response-on-cluster-failure scenario.
The reason why the socket approach is also probably the best bet in
production environments is that when you are scaling SSL termination
across multiple haproxy processes it is often the case that you will use
unix sockets anyway to distribute the load, and you can cleverly reuse
those sockets to do what you want.
So you would have haproxy in tcp mode binding to ports 80 and 443,
creating multiple sockets which different processes would bind to to
handle TLS (and handle http load balancing). By cleverly adjusting the
weights you could reuse these sockets for A/B testing purposes. Also,
this allows you to test different TLS ciphers as well (each process can
define their own TLS ciphers).
I should also point out that you can test different haproxy versions if
you split the haproxy version used for the level 4 load balancing and
the haproxy version used for level 7 load balancing. However, that might
be overkill in your setup.
Considering this approach with L4 load balancing -> sockets -> L7 load
balancing works quite nicely at scale, I would recommend it, though
there are a few pitfalls associated with this. Heavy testing of the
HAProxy configuration in staging is recommended or chatting with our
enterprise support to avoid nasty surprises.
Please note that you should consider setting send-proxy-v2 and
accept-proxy respectively to preserve source IP information when using
sockets. Also, tcp-request inspect-delay is your friend (multiples of 3s
are recommended to handle tcp re-transmission + 1 e.g. 4s, 5s etc.).
As for the latency/performance impact of sockets, it would be nice to
measure the overhead and see if something can be optimized, but again,
depending on the scale you're working at, it might be negligible
compared to the elephant in the room (TLS CPU overhead, especially with
RSA).
Also, with sockets, you can disable routing traffic to a backend simply
by toggling the state of the socket, which is pretty neat.
The crux of the matter is that use_backend doesn't allow you to simply
split the traffic in half in an obvious way (e.g. by simply specifying a
keyword), instead, you have to define an ACL that would to that. You
could do that by setting a special header or a cookie in your
application code which would allow you to construct an ACL rule to split
the traffic on.
An useful feature would be to modify the use_backend syntax to allow for
easier splitting across multiple backends. It might be possible that Lua
can achieve this. Can't make any promises, but I will investigate
further into this if I have some spare time.
Another approach would be to add a feature that automatically rebalances
all weights if a server is added/removed (so you would just use one big
backend): However, this is also achievable by appropriately wrapping the
HAProxy API. Might be less painful than the alternatives.
Another approach if you are working with multiple haproxy instances
would be to deploy different haproxy configurations to different servers
(one haproxy server exclusively servers A traffic, another one B traffic).
To conclude, there are multiple approaches that might yield good results
with HAProxy, and some good features for better testing could be
developed, but one should consider if this should be done by HAProxy at all.
Your application might be able to make the decision instead in an
environment you are more comfortable with: the programming language used
by your team.
If you do A/B testing in your application, you can use the entire server
fleet
instead of sacrificing 50% of your servers as guinea pigs.
What happens when e.g. the entire B backend starts serving 500 responses?
The load on your A backend will double. Are you sure you have the
capacity for that?
Contrast that to implementing the change via a feature flag: if anything
goes wrong with "B" you change one value in e.g. your database, and the
entire server fleet switches to using "A".
Without knowing more about the environment you are performing A/B
testing (production, development, staging), the scale you are working
at, and which part of the stack you are A/B testing, it is difficult to
come up with a recommendation.
Let me know if you have any questions.
Best regards,
Bruno Henc
On 2/11/19 10:15 PM, James Root wrote:
Hey Aleks,
Thank you for the reply, I should have included my version. I am
currently using HAProxy 1.8, but moving up a version is a possibility.
I understand what your example is doing, but it has the same issue my
original example has I think, that I have to have one unix socket per
cluster. In my case, "cluster" is just a small collection of servers
with the same service, but there could be dozens of these clusters.
In our setup, an inbound request gets routed to the correct backend
based on the host header (in my original example, this backend would
be "haproxy-test"). But then I effectively want to A/B test between
the two clusters that can serve this backend. I could put ever server
in this one backend, with the proper weights, but that isn't exactly
what I am looking for. Ideally, I would like if one cluster goes out
completely that 503s get returned for any requests that would normally
get round robined to that cluster. The only way I could find to
actually enforce weighting between two clusters was to forward the
request through a socket to a new "frontend" (functionally this is the
same as running a proxy instance per cluster). This seems to work, but
I am looking for a way to do it without opening up a large amount of
unix sockets.
On Wed, Feb 6, 2019 at 11:43 AM Aleksandar Lazic <[email protected]
<mailto:[email protected]>> wrote:
Hi James.
Am 06.02.2019 um 16:16 schrieb James Root:
> Hi All,
>
> I am doing some research and have not really found a great way
to configure
> HAProxy to get the desired results. The problem I face is that I
a service
> backed by two separate collections of servers. I would like to
split traffic
> between these two clusters (either using percentages or
weights). Normally, I
> would configure a single backend and calculate my weights to get
the desired
> effect. However, for my use case, the list of servers can be
update dynamically
> through the API. To maintain correct weighting, I would then have to
> re-calculate the weights of every entry to maintain a correct
balance.
>
> An alternative I found was to do the following in my
configuration file:
>
> backend haproxy-test
> balance roundrobin
> server cluster1 [email protected] weight 90
> server cluster2 [email protected] weight 10
>
> listen cluster1
> bind [email protected]
> balance roundrobin
> server s1 127.0.0.1:8081 <http://127.0.0.1:8081>
<http://127.0.0.1:8081>
>
> listen cluster2
> bind [email protected]
> balance roundrobin
> server s1 127.0.0.1:8082 <http://127.0.0.1:8082>
<http://127.0.0.1:8082>
> server s2 127.0.0.1:8083 <http://127.0.0.1:8083>
<http://127.0.0.1:8083>
>
> This works, but is a bit nasty because it has to take another
round trip through
> the kernel. Ideally, there would be a way to accomplish this
without having to
> open unix sockets, but I couldn't find any examples or any leads
in the haproxy
> docs.
>
> I was wondering if anyone on this list had any ideas to
accomplish this without
> using extra unix sockets? Or an entirely different way to get
the same effect?
Well as we don't know which version of HAProxy do you use I will
suggest you a
solution based on 1.9.
I would try to use the set-priority-* feature
https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#4.2-http-request%20set-priority-class
https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#4.2-http-request%20set-priority-offset
https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#7.3.2-prio_class
https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#7.3.2-prio_offset
https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#7.3.3-src
I would try the following, untested but I think you get the idea.
frontend clusters
bind [email protected]
bind [email protected]
balance roundrobin
# I'm not sure if src works with unix sockets like this
# maybe you need to remove the unix@ part.
acl src-cl1 src [email protected]
acl src-cl2 src [email protected]
http-request set-priority-class -10s if src-cl1
http-request set-priority-class +10s if src-cl2
# http-request set-priority-offset 5s if LOGO
# http-request set-priority-offset 5s if LOGO
use_backend cluster1 if priority-class < 5
use_backend cluster2 if priority-class > 5
backend cluster1
server s1 127.0.0.1:8081 <http://127.0.0.1:8081>
backend cluster2
server s1 127.0.0.1:8082 <http://127.0.0.1:8082>
server s2 127.0.0.1:8083 <http://127.0.0.1:8083>
There are a lot of fetching functions so maybe you find a better
solution with
another fetch function as I don't know your application.
https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#7
In case you haven't seen it there is also a management interface
for haproxy.
https://cbonte.github.io/haproxy-dconv/1.9/management.html#9.3
https://www.haproxy.com/blog/dynamic-configuration-haproxy-runtime-api/
> Thanks,
> James Root
Regards
Aleks
