Hi Mark,
Le 15/02/2014 21:28, Mark Ruys a écrit :
Hi,
I've read the blog
http://blog.exceliance.fr/2014/01/17/emulating-activepassing-application-clustering-with-haproxy/
which
describes how to prevent an automatic fail back in case of a a fail over
in a active/passive setting. This is important for us as we want to be
sure that MySQL replication is fully recovered before the failed master
gets active again.
It's confusing because the examples you provided are for HTTP servers
and you're talking about MySQL replication. In many cases, you'll manage
it differently. Well, I'll use HTTP too for my example but will try to
describe a solution that works for mysql.
In a typically HAProxy setting (at least in our case), we have backend
servers serving a few IP-adresses (each SSL certificate needs its own IP
address). So the frontend binds to a few IP-addresses. But the stick
table can only hold a single dst IP-address. Shouldn't the size of stick
table be at least at big as the number of addresses the frontend binds to?
Then I have another problem. In some unfortunate scenarios the network
gets unreliable. So it could be the case that server s1 is unreachable
and automatic fail over to s2 occurs. This is sticky, so when s2 is
reachable again, no automatic fail back happens, just as we want. But
then s2 gets unreachable. In this case an automatic fail back will
start! This is against the prerequisite that a fail back should /only/
occur by manual intervention. So wouldn't it be safer to define the
backend like this:
backend bk_app
stick-table type ip size 1 peers LB
stick on dst
server s1 10.0.0.1:80 check rise 9999999
server s2 10.0.0.2:80 check backup
Then we've got 100 days time to review the situation and force a manual
fail back when I'm ready (by reloading the haproxy service).
I think that for such a critical scenario, it can be a good idea to
split servers into 2 backends.
Example :
backend active
server s-active 10.0.0.1:80 check
backend passive
server s-passive 10.0.0.2:80
Then a frontend can elect the backend depending on the "active" backend
state :
frontend cluster :80
acl PASSIVE nbsrv(active) eq 0
use_backend passive if PASSIVE
default_backend active
This will (almost) simulate the "backup" behaviour, without some
features such as redispatch. But in a replication scenario, I prefer a
connection error than a redispatch that will write to the wrong server.
Now, it's time to have a sticky cluster. For that :
1. Add 1 stick table in the "passive" backend,
2. Add a new condition in the frontend to check if there's an entry or
not (this is where I introduce "always_true");
3. Peers are used to remember the states on reload
4. Also add a stats socket to allow a manual reset (switch back to the
active server).
Example :
global
daemon
pidfile /var/run/tmp/haproxy.pid
stats socket /var/run/haproxy.sock
peers LB
peer LB1 10.0.0.98:1234
peer LB2 10.0.0.99:1234
defaults
timeout client 20s
timeout server 20s
timeout connect 4s
frontend cluster :80
acl PASSIVE nbsrv(active) eq 0
acl PASSIVE table_cnt(passive) gt 0
use_backend passive if PASSIVE
default_backend active
backend active
server s-active 10.0.0.1:80 check
backend passive
stick-table type integer size 1 peers LB
stick on always_true
server s-passive 10.0.0.2:80 # is it necessary to check the passive
one ?
From this example, the default backend will be the active one.
Once "s-active" is DOWN or an entry exists in the "passive" table, the
"passive" backend will be used.
To switch back to the "active" backend :
# echo "clear table passive" | socat stdio /var/run/haproxy.sock
--
Cyril Bonté
