Re: SYN dropping at reload time

[Willy Tarreau]

On Tue, Aug 09, 2016 at 05:06:15PM +0200, Aleksandar Lazic wrote:
> 
> 
> Am 09-08-2016 17:03, schrieb Lukas Tribus:
> > Am 09.08.2016 um 16:39 schrieb Aleksandar Lazic:
> > > 
> > > > On Linux yes, because there is a race condition:
> > > > 
> > > > http://lwn.net/Articles/542866/
> > > > http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html
> > > 
> > > Thanks.
> > > 
> > > So every Software on Linux should have this issue right?
> > > 
> > > I'm one of the lucky person who have not faced this problem ;-).
> > > 
> > > But the same issue can happen with nginx, apache httpd, uwsgi, and
> > > so on also, right.
> > 
> > That depends on the situation.
> > 
> > Nginx uses a master and worker processes for example. If you only
> > change the configuration, the master process may not get restarted
> > (only workers are restarted), so the race is irrelevant. How exactly
> > nginx handles the binary upgrade on Linux - I don't know.
> 
> Yes that's true. that's a different concept.

Note that it's not a matter of process but of socket. In SO_REUSEPORT mode
we have multiple sockets for a same ip:port. So packets are delivered into
individual sockets. When the last user of a socket closes it, the socket is
closed, and a reset is sent for all packets in its queue.

The only way to avoid this would be to have a master/worker model where the
master is never restarted, deals with sockets and passes them to the other
processes over unix sockets. A first step for this was developped by Simon
a few years ago but he probably faced all the architecture limitations at
once, resulting in various stuff like segfaults on reloads and I don't
remember what!

The exact model needed in fact is not a master/worker but a socket server
(which the master often is in such a model). But to a certain extent we
could very well imagine an external process used as a socket server and
being reconfigured upon every reload, delivering sockets and keeping
refcounts on them.

This is something that could be worth reconsidering for 1.8 if people are
willing to work on this, but it's a huge effort due to a lot of history
accumulated in this dirty area to be honnest.

Willy

Need Help : Haproxy as server with CA signed cert to fetch self-signed client certificate

[Deepak Agarwal]

Hi Baptiste,
As discussed, Please help with inputs on the following.
http://discourse.haproxy.org/t/haproxy-as-server-with-ca-signed-cert-to-fetch-self-signed-client-certificate/551
http://discourse.haproxy.org/t/how-to-fetch-ssl-subjectaltname-san-extension-data-in-haproxy/539
 
Thanks,
Deepak
  

Re: SYN dropping at reload time

[Aleksandar Lazic]

Am 09-08-2016 17:03, schrieb Lukas Tribus:

Am 09.08.2016 um 16:39 schrieb Aleksandar Lazic:



On Linux yes, because there is a race condition:

http://lwn.net/Articles/542866/
http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html


Thanks.

So every Software on Linux should have this issue right?

I'm one of the lucky person who have not faced this problem ;-).

But the same issue can happen with nginx, apache httpd, uwsgi, and so 
on also, right.


That depends on the situation.

Nginx uses a master and worker processes for example. If you only
change the configuration, the master process may not get restarted
(only workers are restarted), so the race is irrelevant. How exactly
nginx handles the binary upgrade on Linux - I don't know.


Yes that's true. that's a different concept.

Cheers Aleks

Re: SYN dropping at reload time

[Lukas Tribus]

Am 09.08.2016 um 16:39 schrieb Aleksandar Lazic:



On Linux yes, because there is a race condition:

http://lwn.net/Articles/542866/
http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html 



Thanks.

So every Software on Linux should have this issue right?

I'm one of the lucky person who have not faced this problem ;-).

But the same issue can happen with nginx, apache httpd, uwsgi, and so 
on also, right.


That depends on the situation.

Nginx uses a master and worker processes for example. If you only change 
the configuration, the master process may not get restarted (only 
workers are restarted), so the race is irrelevant. How exactly nginx 
handles the binary upgrade on Linux - I don't know.


Testing is the only way to find out for sure (the yelp blog has some 
examples about how to test this with ab).



Lukas

Re: SYN dropping at reload time

[Aleksandar Lazic]

Hi Lukas.

Am 09-08-2016 16:13, schrieb Lukas Tribus:

Hi Aleksandar,


Am 09.08.2016 um 13:42 schrieb Aleksandar Lazic:

I know it's not the latest one ;-/.
Is there really a possibility that at soft-reload incoming request are 
failed?


On Linux yes, because there is a race condition:

http://lwn.net/Articles/542866/
http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html


Thanks.

So every Software on Linux should have this issue right?

I'm one of the lucky person who have not faced this problem ;-).

But the same issue can happen with nginx, apache httpd, uwsgi, and so on 
also, right.


Thanks for sharing this information ;-)


OpenBSD does not suffer from this problem (but it also does not
load-balance between processes).

Linux 4.6 allows to attach BPF filters to the TCP sockets of a
SO_REUSEPORT "group", maybe that way we can drain the socket of the
"closing process" gracefully in the future.


Cheers Aleks

Re: SYN dropping at reload time

[Lukas Tribus]

Hi Aleksandar,


Am 09.08.2016 um 13:42 schrieb Aleksandar Lazic:

I know it's not the latest one ;-/.
Is there really a possibility that at soft-reload incoming request are 
failed?



On Linux yes, because there is a race condition:

http://lwn.net/Articles/542866/
http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html


OpenBSD does not suffer from this problem (but it also does not 
load-balance between processes).



Linux 4.6 allows to attach BPF filters to the TCP sockets of a 
SO_REUSEPORT "group", maybe that way we can drain the socket of the 
"closing process" gracefully in the future.




Regards,

Lukas

SYN dropping at reload time

[Aleksandar Lazic]

Hi all.

I just seen in the OpenShift repo the following comment.

https://github.com/openshift/origin/blob/master/images/router/haproxy/reload-haproxy#L61-L63

###
 # We install the syn eater so that connections that come in during the 
restart don't

 # go onto the wrong socket, which is then closed.
###

The reload is done in line 84

https://github.com/openshift/origin/blob/master/images/router/haproxy/reload-haproxy#L84

They use

###
/usr/sbin/haproxy -vv
HA-Proxy version 1.5.14 2015/07/02
Copyright 2000-2015 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -DTCP_USER_TIMEOUT=18
  OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 
USE_PCRE=1


Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 
200


Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT 
IPV6_TRANSPARENT IP_FREEBIND


Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.
###

I know it's not the latest one ;-/.
Is there really a possibility that at soft-reload incoming request are 
failed?


Thanks for info.

Cheers Aleks

Re: 51Degrees Trie Update Patch

[Willy Tarreau]

Hi Ben,

On Mon, Aug 08, 2016 at 03:25:04PM +, Ben Shillito wrote:
> Hi Willy,
>
> Yes I agree, this is a problem.
> I have attached a patch with a change to the readme which explains that that
> the correct version must be pulled if using <=1.6.

(...)
+and checkout the latest compatible version (this step is only neccessary when
+using version 1.6 or earlier) :
+
+git checkout tags/v3.2.5.3
+

Well, if we mention a tag in the readme for stable branches, that means
that this code will never ever receive fixes anymore ? That doesn't sound
very wise, as I guess this doc will have to change next time you face a
bug. Why don't you simply create a branch named with the previous API ?
That way when you want to merge fixes you simply put them into that branch
and you don't have to fear about the many people stuck to your old tag.

Regards,
Willy

Re: Haproxy 1.6.7 segmentation fault under load

[Willy Tarreau]

On Mon, Aug 08, 2016 at 09:38:14PM +, James Hartshorn wrote:
> Thank you so much for your investigation!
> 
> 
> We have been running the patch in high volume testing all day and so far it 
> is working fine with no segfaults.

Excellent, thanks James for this pleasant feedback!

Willy