Re: Easy haproxy redundancy

[Shawn Heisey]

On 8/27/2015 10:22 PM, Nathan Williams wrote:
> We have 2 OpenStack VMs with IPs on the internal network, a keepalived
> -managed VIP on the internal network that's added to each VMs allowed
> -address-pairs in neutron, and a floating IP from the external network
> mapped to the internal VIP (OpenStack floating IP is just a SNAT/DNAT).
> Depending on your environment, that's probably not super relevant, but
> it's essential to being able to have a public VIP under neutron, so I
> thought I'd mention it.

Lots of things I've never touched. My load balancers are behind the
firewall, but use actual public IP addresses on the outside interface,
with a private address on the inside interface. One of the VIPs is a
gateway on the private network for the back end servers. No NAT required.

> keepalived.conf[0]: keepalived manages the vip, runs scored health
> checks: the master has the highest score. keepalived also handles VIP
> migration and gratuitous arp to announce the change on failover.
> 
> role-check.sh[1]: fail the master if it doesn't have the VIP. not
> strictly necessary, but we're paranoid.
> 
> keepalive-notify.sh[2]: record history of state changes, last row used
> by role-check.sh to determine current state.

These are currently unfamiliar concepts, perhaps they will become
familiar once I start digging into keepalived.

> It's been really stable over the last 8 months we've been running it;
> failover works really cleanly like you'd expect, there's been no
> unexplainable failovers we've run into so far, no failure to failover
> when it should. Without using LVS to do the actual load-balancing,
> there's no keepalived-associated tools I'm aware of to let you inspect
> the cluster state (as compared to pacemaker `crm_mon -Af`). It doesn't
> really matter with this simple cluster configuration, though; the
> master's the one with the VIP, so `ip addr show` tells you which one's
> the master, and the notify script-generated log file confirms it.
> 
> You can manually force a failover by stopping the keepalived service on
> the master or adding an exit 1 to the top of the role-check.sh script
> (adjusting the scoring). Ditto for putting the standby into maintenance
> mode, you just ensure it can't end up with a higher score than the
> master.
> 
> One thing to be aware of is that keepalived uses multicast by default,
> so you want to make sure every cluster uses a unique router-id, or your
> clusters might interfere with each other.

I'm using multicast now with the corosync that operates under pacemaker,
so this is not an unusual requirement.  I just choose a different
multicast address for each cluster.

Thanks for the info.

Thanks,
Shawn

Re: Easy haproxy redundancy

[Nathan Williams]

On Fri, 2015-08-28 at 01:25 +, Nathan Williams wrote:
> Yeah, keepalived handles the gratuitous arp on failover, it works
> nicely. I do miss the admin tools for pacemaker though. I'm AFK, but
> I'll write up a full explanation of our HA setup when I'm back at a
> PC.
> Cheers,
> Nathan
> 

Okay, here's the details on how we're doing this with keepalived.

We have 2 OpenStack VMs with IPs on the internal network, a keepalived
-managed VIP on the internal network that's added to each VMs allowed
-address-pairs in neutron, and a floating IP from the external network
mapped to the internal VIP (OpenStack floating IP is just a SNAT/DNAT).
Depending on your environment, that's probably not super relevant, but
it's essential to being able to have a public VIP under neutron, so I
thought I'd mention it.

We set the sysctl to enable binding to an address that we don't have.
There's some other LB sysctl settings we set on the LB, but they're
tunings, and not essential to the HA configuration.

`sysctl net.ipv4.ip_nonlocal_bind=1`

keepalived.conf[0]: keepalived manages the vip, runs scored health
checks: the master has the highest score. keepalived also handles VIP
migration and gratuitous arp to announce the change on failover.

role-check.sh[1]: fail the master if it doesn't have the VIP. not
strictly necessary, but we're paranoid.

keepalive-notify.sh[2]: record history of state changes, last row used
by role-check.sh to determine current state.

It's been really stable over the last 8 months we've been running it;
failover works really cleanly like you'd expect, there's been no
unexplainable failovers we've run into so far, no failure to failover
when it should. Without using LVS to do the actual load-balancing,
there's no keepalived-associated tools I'm aware of to let you inspect
the cluster state (as compared to pacemaker `crm_mon -Af`). It doesn't
really matter with this simple cluster configuration, though; the
master's the one with the VIP, so `ip addr show` tells you which one's
the master, and the notify script-generated log file confirms it.

You can manually force a failover by stopping the keepalived service on
the master or adding an exit 1 to the top of the role-check.sh script
(adjusting the scoring). Ditto for putting the standby into maintenance
mode, you just ensure it can't end up with a higher score than the
master.

One thing to be aware of is that keepalived uses multicast by default,
so you want to make sure every cluster uses a unique router-id, or your
clusters might interfere with each other.

Anyways, hope that helps! Feel free to ask if you have any add'l
questions :)

Regards,

Nathan

[0]: https://gist.github.com/nathwill/2463002f342cc75ae6b0
[1]: https://gist.github.com/nathwill/5475ff3b891c7f2b44b3
[2]: https://gist.github.com/nathwill/ac75957052bd75597780

On Thu, Aug 27, 2015, 6:11 PM Shawn Heisey 
> wrote:
> > On 8/27/2015 6:52 PM, Nathan Williams wrote:
> > > There's a sysctl for that, net.ipv4.ip_nonlocal_bind.
> > 
> > Interesting.  That's one I had never seen before.  I would assume
> > that
> > the OS does this intelligently so that when the IP address *does*
> > suddenly appear at a later time, the application works seamlessly.
> > That's something I will have to test.
> > 
> > I might need to rethink my redundancy design with this new bit of
> > knowledge.  I have seen a number of incidents where pacemaker froze
> > and
> > I had no idea there was a problem until I did maintenance on the
> > standby
> > server (either rebooting it or stopping pacemaker) and the online
> > host
> > didn't notice it going down.  Everything kept working, but the
> > systems
> > were in a state where no failover would have occurred in the event
> > of a
> > failure.  I bet keepalived is a lot more stable than pacemaker.
> > 
> > Thanks,
> > Shawn
> > 
> > 
> > 

Re: Easy haproxy redundancy

[Nathan Williams]

Yeah, keepalived handles the gratuitous arp on failover, it works nicely. I
do miss the admin tools for pacemaker though. I'm AFK, but I'll write up a
full explanation of our HA setup when I'm back at a PC.

Cheers,

Nathan

On Thu, Aug 27, 2015, 6:11 PM Shawn Heisey  wrote:

> On 8/27/2015 6:52 PM, Nathan Williams wrote:
> > There's a sysctl for that, net.ipv4.ip_nonlocal_bind.
>
> Interesting.  That's one I had never seen before.  I would assume that
> the OS does this intelligently so that when the IP address *does*
> suddenly appear at a later time, the application works seamlessly.
> That's something I will have to test.
>
> I might need to rethink my redundancy design with this new bit of
> knowledge.  I have seen a number of incidents where pacemaker froze and
> I had no idea there was a problem until I did maintenance on the standby
> server (either rebooting it or stopping pacemaker) and the online host
> didn't notice it going down.  Everything kept working, but the systems
> were in a state where no failover would have occurred in the event of a
> failure.  I bet keepalived is a lot more stable than pacemaker.
>
> Thanks,
> Shawn
>
>
>

Re: Easy haproxy redundancy

[Shawn Heisey]

On 8/27/2015 6:52 PM, Nathan Williams wrote:
> There's a sysctl for that, net.ipv4.ip_nonlocal_bind.

Interesting.  That's one I had never seen before.  I would assume that
the OS does this intelligently so that when the IP address *does*
suddenly appear at a later time, the application works seamlessly.
That's something I will have to test.

I might need to rethink my redundancy design with this new bit of
knowledge.  I have seen a number of incidents where pacemaker froze and
I had no idea there was a problem until I did maintenance on the standby
server (either rebooting it or stopping pacemaker) and the online host
didn't notice it going down.  Everything kept working, but the systems
were in a state where no failover would have occurred in the event of a
failure.  I bet keepalived is a lot more stable than pacemaker.

Thanks,
Shawn

Re: Easy haproxy redundancy

[Nathan Williams]

There's a sysctl for that, net.ipv4.ip_nonlocal_bind.

On Thu, Aug 27, 2015, 5:49 PM Shawn Heisey  wrote:

> On 8/24/2015 12:06 PM, Dennis Jacobfeuerborn wrote:
> > There is no need to run a full Pacemaker stack. Just run HAProxy on both
> > nodes and manage the virtual ips using keepalived.
>
> All of my bind statements are applied to specific ip addresses, not
> 0.0.0.0.
>
> If you try to start haproxy on a machine that is missing the address(es)
> that you are binding to (which describes the standby server in a
> redundant pair), it won't start.  Public IP addresses redacted in the
> following partial log:
>
> root@lb4:~# service haproxy start ; service haproxy stop
>  * Starting haproxy haproxy
> [ALERT] 238/183842 (32404) : Starting frontend fe-spark-80: cannot bind
> socket [RE.DAC.TED.78:80]
> [ALERT] 238/183842 (32404) : Starting frontend fe-spark-443: cannot bind
> socket [RE.DAC.TED.78:443]
>
> This is why I run redundant haproxy with a full pacemaker stack that
> starts haproxy and the gratuitous arps *after* the address resources
> have started.
>
> Thanks,
> Shawn
>
>
>

Re: Easy haproxy redundancy

[Shawn Heisey]

On 8/24/2015 12:06 PM, Dennis Jacobfeuerborn wrote:
> There is no need to run a full Pacemaker stack. Just run HAProxy on both
> nodes and manage the virtual ips using keepalived.

All of my bind statements are applied to specific ip addresses, not 0.0.0.0.

If you try to start haproxy on a machine that is missing the address(es)
that you are binding to (which describes the standby server in a
redundant pair), it won't start.  Public IP addresses redacted in the
following partial log:

root@lb4:~# service haproxy start ; service haproxy stop
 * Starting haproxy haproxy
[ALERT] 238/183842 (32404) : Starting frontend fe-spark-80: cannot bind
socket [RE.DAC.TED.78:80]
[ALERT] 238/183842 (32404) : Starting frontend fe-spark-443: cannot bind
socket [RE.DAC.TED.78:443]

This is why I run redundant haproxy with a full pacemaker stack that
starts haproxy and the gratuitous arps *after* the address resources
have started.

Thanks,
Shawn

Re: Easy haproxy redundancy

[Dennis Jacobfeuerborn]

There is no need to run a full Pacemaker stack. Just run HAProxy on both
nodes and manage the virtual ips using keepalived.

Regards,
  Dennis

On 08/24/2015 06:09 PM, Kobus Bensch wrote:
> Hi
> 
> I have redundant haproxy servers on my environment. We use corosync and
> pacemaker that manages the HA and then have HAproxy run on the HA domain
> controller.
> 
> 
> 
> On 21/08/2015 15:51, Jeff Palmer wrote:
>> I've done exactly this.   Amazon AWS has a DNS service called route53.
>> Route53 has built in "health checks"
>>
>>
>> What I did was setup the 2 haproxy nodes as A records, with a
>> healthcheck and a TTL of 30 seconds.
>> If one of the haproxy nodes failed the health check twice, route53
>> would remove it from the RR record set.
>>
>> While not a *perfect* solution,  60 seconds of an unresponsive haproxy
>> node was better for my use-case,  than say..  monitoring it, and
>> manually updating DNS, or moving IP's to new nodes,  or whatever else.
>> maybe that would suffice for you as well?
>>
>>
>> If you already have a DNS provider you like,  you could do like I did,
>> and delegate a subdomain to route53.
>>
>> In my case, I had "example.com" that used another DNS provider.   I
>> delegated "lb.example.com" to amazons route53.   my haproxy nodes were
>> then:
>>
>> fe1.lb.example.com
>> fe2.lb.example.com
>> fe3.lb.example.com
>>
>> records that needed haproxy on front of them,  just became cnames to
>> "lb.example.com"
>>
>>
>> Hope that makes sense!
>>
>>
>>
>>
>> On Fri, Aug 21, 2015 at 10:04 AM, Nikolaos Milas  wrote:
>>> Hello,
>>>
>>> We are setting up a proxy, a haproxy server on CentOS 7, to our mail
>>> services (webmail, smtp, pop3, imap, simple and with STARTTLS, or
>>> SSL/TLS as
>>> appropriate). The load of the services is considered low. All clients
>>> will
>>> be accessing the above services through the new proxy.
>>>
>>> Current goal: To provide redundancy (fail-over) of the haproxy server.
>>>
>>> I have read: http://www.serverphorums.com/read.php?10,255589 which
>>> provides
>>> valuable information, but I would like your opinions, due to the
>>> limitations
>>> we face (see below).
>>>
>>> All our VPS servers are provided free of charge (we are a non-profit
>>> scientific research foundation) by our ISP, but there are limitations:
>>>
>>> - All our servers (DNS, Mail, Web, etc.) are hosted on VPSs (i.e.
>>> they are VMs) on two different data centers (on our ISP's cloud),
>>> i.e. we don't have any local physical servers available
>>> - Each VPS server must have a single (exactly one) distinct
>>> permanent IP Address and a single network interface
>>> - We don't control how each VM is connected to the Internet
>>> - We don't have any SLAs for network or VPS availability
>>>
>>> On the good side, the uptime is very high; we rarely face downtime,
>>> yet, we
>>> need redundancy for the rare occasions when a VM will not be
>>> available due
>>> to hardware or network issues.
>>>
>>> It would be enough for us to be able to use two VMs (each running
>>> haproxy
>>> with identical configuration), one on each of the two data centers,
>>> as an
>>> active/passive pair.
>>>
>>> However, under the above circumstances, I find it difficult to use the
>>> usually suggested solutions of keepalived, heartbeat, pacemaker (and any
>>> similar software which causes IP Address changes). A common DNS name
>>> with
>>> two A records is not a reliable solution.
>>>
>>> So, could you please provide some opinions/advice on how to move on
>>> with our
>>> available resources?
>>>
>>> Thanks in advance,
>>> Nick
>>>
>>
>>
> 

Re: Easy haproxy redundancy

[Kobus Bensch]

Hi

I have redundant haproxy servers on my environment. We use corosync and 
pacemaker that manages the HA and then have HAproxy run on the HA domain 
controller.




On 21/08/2015 15:51, Jeff Palmer wrote:

I've done exactly this.   Amazon AWS has a DNS service called route53.
Route53 has built in "health checks"


What I did was setup the 2 haproxy nodes as A records, with a
healthcheck and a TTL of 30 seconds.
If one of the haproxy nodes failed the health check twice, route53
would remove it from the RR record set.

While not a *perfect* solution,  60 seconds of an unresponsive haproxy
node was better for my use-case,  than say..  monitoring it, and
manually updating DNS, or moving IP's to new nodes,  or whatever else.
maybe that would suffice for you as well?


If you already have a DNS provider you like,  you could do like I did,
and delegate a subdomain to route53.

In my case, I had "example.com" that used another DNS provider.   I
delegated "lb.example.com" to amazons route53.   my haproxy nodes were
then:

fe1.lb.example.com
fe2.lb.example.com
fe3.lb.example.com

records that needed haproxy on front of them,  just became cnames to
"lb.example.com"


Hope that makes sense!




On Fri, Aug 21, 2015 at 10:04 AM, Nikolaos Milas  wrote:

Hello,

We are setting up a proxy, a haproxy server on CentOS 7, to our mail
services (webmail, smtp, pop3, imap, simple and with STARTTLS, or SSL/TLS as
appropriate). The load of the services is considered low. All clients will
be accessing the above services through the new proxy.

Current goal: To provide redundancy (fail-over) of the haproxy server.

I have read: http://www.serverphorums.com/read.php?10,255589 which provides
valuable information, but I would like your opinions, due to the limitations
we face (see below).

All our VPS servers are provided free of charge (we are a non-profit
scientific research foundation) by our ISP, but there are limitations:

- All our servers (DNS, Mail, Web, etc.) are hosted on VPSs (i.e.
they are VMs) on two different data centers (on our ISP's cloud),
i.e. we don't have any local physical servers available
- Each VPS server must have a single (exactly one) distinct
permanent IP Address and a single network interface
- We don't control how each VM is connected to the Internet
- We don't have any SLAs for network or VPS availability

On the good side, the uptime is very high; we rarely face downtime, yet, we
need redundancy for the rare occasions when a VM will not be available due
to hardware or network issues.

It would be enough for us to be able to use two VMs (each running haproxy
with identical configuration), one on each of the two data centers, as an
active/passive pair.

However, under the above circumstances, I find it difficult to use the
usually suggested solutions of keepalived, heartbeat, pacemaker (and any
similar software which causes IP Address changes). A common DNS name with
two A records is not a reliable solution.

So, could you please provide some opinions/advice on how to move on with our
available resources?

Thanks in advance,
Nick






--
Kobus Bensch Trustpay Global LTD email signature Kobus Bensch
Senior Systems Administrator
Address:  22 & 24 | Frederick Sanger Road | Guildford | Surrey | GU2 7YD
DDI:  0207 871 3958
Tel:  0207 871 3890
Email: kobus.ben...@trustpayglobal.com 



--


Trustpay Global Limited is an authorised Electronic Money Institution 
regulated by the Financial Conduct Authority registration number 900043. 
Company No 07427913 Registered in England and Wales with registered address 
130 Wood Street, London, EC2V 6DL, United Kingdom.


For further details please visit our website at www.trustpayglobal.com.

The information in this email and any attachments are confidential and 
remain the property of Trustpay Global Ltd unless agreed by contract. It is 
intended solely for the person to whom or the entity to which it is 
addressed. If you are not the intended recipient you may not use, disclose, 
copy, distribute, print or rely on the content of this email or its 
attachments. If this email has been received by you in error please advise 
the sender and delete the email from your system. Trustpay Global Ltd does 
not accept any liability for any personal view expressed in this message.

Re: Easy haproxy redundancy

[Jeff Palmer]

I've done exactly this.   Amazon AWS has a DNS service called route53.
Route53 has built in "health checks"


What I did was setup the 2 haproxy nodes as A records, with a
healthcheck and a TTL of 30 seconds.
If one of the haproxy nodes failed the health check twice, route53
would remove it from the RR record set.

While not a *perfect* solution,  60 seconds of an unresponsive haproxy
node was better for my use-case,  than say..  monitoring it, and
manually updating DNS, or moving IP's to new nodes,  or whatever else.
maybe that would suffice for you as well?


If you already have a DNS provider you like,  you could do like I did,
and delegate a subdomain to route53.

In my case, I had "example.com" that used another DNS provider.   I
delegated "lb.example.com" to amazons route53.   my haproxy nodes were
then:

fe1.lb.example.com
fe2.lb.example.com
fe3.lb.example.com

records that needed haproxy on front of them,  just became cnames to
"lb.example.com"


Hope that makes sense!




On Fri, Aug 21, 2015 at 10:04 AM, Nikolaos Milas  wrote:
> Hello,
>
> We are setting up a proxy, a haproxy server on CentOS 7, to our mail
> services (webmail, smtp, pop3, imap, simple and with STARTTLS, or SSL/TLS as
> appropriate). The load of the services is considered low. All clients will
> be accessing the above services through the new proxy.
>
> Current goal: To provide redundancy (fail-over) of the haproxy server.
>
> I have read: http://www.serverphorums.com/read.php?10,255589 which provides
> valuable information, but I would like your opinions, due to the limitations
> we face (see below).
>
> All our VPS servers are provided free of charge (we are a non-profit
> scientific research foundation) by our ISP, but there are limitations:
>
>- All our servers (DNS, Mail, Web, etc.) are hosted on VPSs (i.e.
>they are VMs) on two different data centers (on our ISP's cloud),
>i.e. we don't have any local physical servers available
>- Each VPS server must have a single (exactly one) distinct
>permanent IP Address and a single network interface
>- We don't control how each VM is connected to the Internet
>- We don't have any SLAs for network or VPS availability
>
> On the good side, the uptime is very high; we rarely face downtime, yet, we
> need redundancy for the rare occasions when a VM will not be available due
> to hardware or network issues.
>
> It would be enough for us to be able to use two VMs (each running haproxy
> with identical configuration), one on each of the two data centers, as an
> active/passive pair.
>
> However, under the above circumstances, I find it difficult to use the
> usually suggested solutions of keepalived, heartbeat, pacemaker (and any
> similar software which causes IP Address changes). A common DNS name with
> two A records is not a reliable solution.
>
> So, could you please provide some opinions/advice on how to move on with our
> available resources?
>
> Thanks in advance,
> Nick
>



-- 
Jeff Palmer
https://PalmerIT.net