Re: Reloading haproxy without dropping connections

[Pedro Mata-Mouros]

Hi,

Haven’t had the chance to implement this yet, but maybe these links can get you 
started:

http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html
http://inside.unbounce.com/product-dev/haproxy-reloads/

It’d be cool to have a sort of “officially endorsed” way of achieving this.

Best,

Pedro.



> On 22 Jan 2016, at 00:38, CJ Ess  wrote:
> 
> One of our sore points with HAProxy has been that when we do a reload there 
> is a ~100ms gap where neither the old or new HAproxy processes accept any 
> requests. See attached graphs. I assume that during this time any connections 
> received to the port are dropped. Is there anything we can do so that the old 
> process keeps accepting requests until the new process is completely 
> initialized and starts accepting connections on its own?
> 
> I've looked into fencing the restart with iptable commands to blackhole TCP 
> SYNs, and I've looked into the huptime utility though I'm not sure 
> overloading libc functions is the best approach long term. Any other 
> solutions? 
> 
> 
> 
> 
> 
> 

Re: Reloading haproxy without dropping connections

[CJ Ess]

The yelp solution I can't do because it requires a newer kernel then I have
access to, but the unbounce solution is interesting, I may be able to work
up something around that.



On Fri, Jan 22, 2016 at 4:07 AM, Pedro Mata-Mouros  wrote:

> Hi,
>
> Haven’t had the chance to implement this yet, but maybe these links can
> get you started:
>
>
> http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html
> http://inside.unbounce.com/product-dev/haproxy-reloads/
>
> It’d be cool to have a sort of “officially endorsed” way of achieving this.
>
> Best,
>
> Pedro.
>
>
>
> On 22 Jan 2016, at 00:38, CJ Ess  wrote:
>
> One of our sore points with HAProxy has been that when we do a reload
> there is a ~100ms gap where neither the old or new HAproxy processes accept
> any requests. See attached graphs. I assume that during this time any
> connections received to the port are dropped. Is there anything we can do
> so that the old process keeps accepting requests until the new process is
> completely initialized and starts accepting connections on its own?
>
> I've looked into fencing the restart with iptable commands to blackhole
> TCP SYNs, and I've looked into the huptime utility though I'm not sure
> overloading libc functions is the best approach long term. Any other
> solutions?
>
>
> 
> 
>
>
>
>

Re: Reloading haproxy without dropping connections

[David Martin]

We use the iptables syn drop method, works fine; the additional 1 sec
in response time for the tiny number of new connections doesn't bother
us as we are not restarting multiple time per hour.

On Fri, Jan 22, 2016 at 11:01 AM, CJ Ess  wrote:
> The yelp solution I can't do because it requires a newer kernel then I have
> access to, but the unbounce solution is interesting, I may be able to work
> up something around that.
>
>
>
> On Fri, Jan 22, 2016 at 4:07 AM, Pedro Mata-Mouros
>  wrote:
>>
>> Hi,
>>
>> Haven’t had the chance to implement this yet, but maybe these links can
>> get you started:
>>
>>
>> http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html
>> http://inside.unbounce.com/product-dev/haproxy-reloads/
>>
>> It’d be cool to have a sort of “officially endorsed” way of achieving
>> this.
>>
>> Best,
>>
>> Pedro.
>>
>>
>>
>> On 22 Jan 2016, at 00:38, CJ Ess  wrote:
>>
>> One of our sore points with HAProxy has been that when we do a reload
>> there is a ~100ms gap where neither the old or new HAproxy processes accept
>> any requests. See attached graphs. I assume that during this time any
>> connections received to the port are dropped. Is there anything we can do so
>> that the old process keeps accepting requests until the new process is
>> completely initialized and starts accepting connections on its own?
>>
>> I've looked into fencing the restart with iptable commands to blackhole
>> TCP SYNs, and I've looked into the huptime utility though I'm not sure
>> overloading libc functions is the best approach long term. Any other
>> solutions?
>>
>>
>> 
>> 
>>
>>
>>
>

question about reloading haproxy

[Michele Mazzucco]

Hello,

I am trying to control a haproxy instance running as a daemon via python: my 
script monitors the status of haproxy (e.g., number of queued requests), 
adds/remove some apache servers and updates the configuration of haproxy.
Currently my script runs as a root. Would it be possible to restart haproxy 
(e.g. by issuing the command /etc/init.d/haproxy restart) with lower 
privileges?
The problem seems to be that I cannot re-bind port 80, unless the script runs 
with root privileges.


Thanks,
Michele

Re: question about reloading haproxy

[Michele Mazzucco]

Hello Brane,

yes, but that is equivalent to running the script as root... the effective user 
ID is 0.


Cheers,
Michele


On Feb 10, 2012, at 11:52 AM, Brane F. Gračnar wrote:

 On 02/10/2012 10:35 AM, Michele Mazzucco wrote:
 Currently my script runs as a root. Would it be possible to restart haproxy 
 (e.g. by issuing the command /etc/init.d/haproxy restart) with lower 
 privileges?
 The problem seems to be that I cannot re-bind port 80, unless the script 
 runs with root privileges.
 
 sudo(8) is your friend.
 
 Brane

Re: question about reloading haproxy

[Brane F. Gračnar]

On 02/10/2012 10:53 AM, Michele Mazzucco wrote:
 Hello Brane,
 
 yes, but that is equivalent to running the script as root... the effective 
 user ID is 0.
 

Nope, it's not :) You can configure sudo to allow specific user to run
only /etc/init.d/haproxy reload without entering password.

Your script can run without any special privileges and is still able to
perform some task which requires root privileges.

Brane

Re: question about reloading haproxy

[Baptiste]

On Fri, Feb 10, 2012 at 10:59 AM, Brane F. Gračnar
brane.grac...@tsmedia.si wrote:
 On 02/10/2012 10:53 AM, Michele Mazzucco wrote:
 Hello Brane,

 yes, but that is equivalent to running the script as root... the effective 
 user ID is 0.


 Nope, it's not :) You can configure sudo to allow specific user to run
 only /etc/init.d/haproxy reload without entering password.

 Your script can run without any special privileges and is still able to
 perform some task which requires root privileges.

 Brane


I agree with Brane :)

RE: reloading haproxy

[Robert Simmons]

I wasn't sure when I created this if it would be of interest to the
community, but please find attached my HAProxy Init script. It supports both
reloading and the fancy green OK output ;-).

Usage:
1. Update the user configuration section of the script with the path's to
your HAProxy binary and configuration file
2. Copy into /etc/init.d and name the file haproxy
3. Set the permissions to Executable (chmod +x haproxy)
4. On a Redhat/Fedora system, configure it for use by running chkconfig
--add haproxy and then to get it to start, chkconfig haproxy on.
5. Then use service haproxy start to run HAProxy.

Hope this helps!

Robert.

-Original Message-
From: Willy Tarreau [mailto:w...@1wt.eu] 
Sent: 21 May 2009 05:33
To: Jeremy
Cc: haproxy@formilux.org
Subject: Re: reloading haproxy

On Wed, May 20, 2009 at 05:03:08PM -0500, Jeremy wrote:
 While we're on the subject, in case anyone else wants to use it (or 
 knows a better way of doing this)... I stuck this in my haproxy init.d 
 scrip (using CentOS/RedHat). I'm not sure how to make it do the fancy 
 green [ OK ] output like start/stop does but that's ok.

I don't know for the colors, but I have comments below

 reload() {
   /usr/local/sbin/$BASENAME -c -q -f /etc/$BASENAME/$BASENAME.cfg

here you use $BASENAME ...

   if [ $? -ne 0 ]; then
 echo Errors found in configuration file, check it with '$BASENAME 
 check'.
 return 1
   fi
   /usr/local/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -sf $(pidof
 haproxy)

...and here haproxy.

   sleep 1

don't do that. If you add sleep, it will only make things work by pure luck
and slow your system down for nothing.

   pidof haproxy  /var/run/$BASENAME.pid

better use -p /var/run/$BASENAME.pid like this :

   /usr/local/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p
/var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid 2/dev/null)

That way haproxy sets the pid(s) file itself and checks that file, it does
not rely on a tricky pidof call nor the presence of a sleep call.

Regards,
Willy




haproxy.init
Description: Binary data

Re: reloading haproxy

[Jeremy]

While we're on the subject, in case anyone else wants to use it (or knows a
better way of doing this)... I stuck this in my haproxy init.d scrip (using
CentOS/RedHat). I'm not sure how to make it do the fancy green [ OK ] output
like start/stop does but that's ok.


reload() {
  /usr/local/sbin/$BASENAME -c -q -f /etc/$BASENAME/$BASENAME.cfg
  if [ $? -ne 0 ]; then
echo Errors found in configuration file, check it with '$BASENAME
check'.
return 1
  fi
  /usr/local/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -sf $(pidof
haproxy)
  sleep 1
  pidof haproxy  /var/run/$BASENAME.pid
}



# See how we were called.
case $1 in
...
  reload)
reload
...

Re: reloading haproxy

[Willy Tarreau]

Hi Adrian,

On Thu, May 14, 2009 at 03:33:39PM +0200, Adrian Moisey wrote:
 Hi
 
 I tried that, also gave the same result.

What is happening is that the new haproxy process asks the old one
to release the ports so that it can bind to them. So there exists
a short period of time (a few hundreds of microseconds) during which
neither the old nor the new process owns the port.

This does not happen if your system supports SO_REUSEPORT (such as
various BSDs, as well as Linux kernels which have my patch applied).
On those systems, the new process can bind to the port eventhough
the old one is already bound, so there is absolutely 0 downtime.

Anyway in production it will not matter at all, because in practice,
when a browser faces a connection abort, it retries after a few
hundreds of milliseconds, when the new process is already in place.
And this will only concern the very few requests which can happen
during the switch.

Last, as John said, please ensure to always use -sf instead of -st
if you want your reload to really remain unnoticed, since -st kills
terminates connections.

Regards,
Willy

RE: reloading haproxy

[John Lauro]

This is what I use to reload:

haproxy -D -f /etc/lb-transparent-slave.cfg -sf $(pidof haproxy)

(Which has pidof lookup process id instead of file it in a file, but that
shouldn't matter.)
The main problem is you are (-st) terminating (aborting) existing
connections instead of (-sf) finishing them.



 -Original Message-
 From: Adrian Moisey [mailto:adr...@careerjunction.co.za]
 Sent: Thursday, May 14, 2009 4:33 AM
 To: haproxy@formilux.org
 Subject: reloading haproxy
 
 Hi
 
 I am currently testing HAProxy for deployment in our live environment.
 I have HAProxy setup to load balance between 4 web servers and I'm
 using
 ab (apache bench) for testing throughput.
 
 I am trying to get the haproxy reloading working, but it doesn't seem
 to
 work.
 
 I start up a few ab's and then run:
 /usr/sbin/haproxy -f /etc/haproxy.cfg -D -p /var/run/haproxy.pid -st
 `cat /var/run/haproxy.pid`
 
 I see the new haproxy take over the old one, but my apache bench fails
 with the following error message:
 apr_socket_recv: Connection reset by peer (104)
 
 
 I thought that the new haproxy would take over the old one without any
 issues, but that doesn't seem to be the case.
 
 Does anyone know how do reload haproxy without affecting the client?
 --
 Adrian Moisey
 Acting Systems Designer | CareerJunction | Better jobs. More often.
 Web: www.careerjunction.co.za | Email: adr...@careerjunction.co.za
 Phone: +27 21 818 8621 | Mobile: +27 82 858 7830 | Fax: +27 21 818 8855
 
 Internal Virus Database is out of date.
 Checked by AVG - www.avg.com
 Version: 8.5.320 / Virus Database: 270.12.10/2088 - Release Date:
 05/05/09 13:07:00

Re: reloading haproxy

[Adrian Moisey]

Hi

I tried that, also gave the same result.


Adrian Moisey
Acting Systems Designer | CareerJunction | Better jobs. More often.
Web: www.careerjunction.co.za | Email: adr...@careerjunction.co.za
Phone: +27 21 818 8621 | Mobile: +27 82 858 7830 | Fax: +27 21 818 8855


John Lauro wrote:

This is what I use to reload:

haproxy -D -f /etc/lb-transparent-slave.cfg -sf $(pidof haproxy)

(Which has pidof lookup process id instead of file it in a file, but that
shouldn't matter.)
The main problem is you are (-st) terminating (aborting) existing
connections instead of (-sf) finishing them.




-Original Message-
From: Adrian Moisey [mailto:adr...@careerjunction.co.za]
Sent: Thursday, May 14, 2009 4:33 AM
To: haproxy@formilux.org
Subject: reloading haproxy

Hi

I am currently testing HAProxy for deployment in our live environment.
I have HAProxy setup to load balance between 4 web servers and I'm
using
ab (apache bench) for testing throughput.

I am trying to get the haproxy reloading working, but it doesn't seem
to
work.

I start up a few ab's and then run:
/usr/sbin/haproxy -f /etc/haproxy.cfg -D -p /var/run/haproxy.pid -st
`cat /var/run/haproxy.pid`

I see the new haproxy take over the old one, but my apache bench fails
with the following error message:
apr_socket_recv: Connection reset by peer (104)


I thought that the new haproxy would take over the old one without any
issues, but that doesn't seem to be the case.

Does anyone know how do reload haproxy without affecting the client?
--
Adrian Moisey
Acting Systems Designer | CareerJunction | Better jobs. More often.
Web: www.careerjunction.co.za | Email: adr...@careerjunction.co.za
Phone: +27 21 818 8621 | Mobile: +27 82 858 7830 | Fax: +27 21 818 8855

Internal Virus Database is out of date.
Checked by AVG - www.avg.com
Version: 8.5.320 / Virus Database: 270.12.10/2088 - Release Date:
05/05/09 13:07:00


.