Re: Increase in sockets in TIME_WAIT with 1.9.x

2019-06-14 Thread Willy Tarreau
Hi again, On Fri, Jun 14, 2019 at 05:25:34AM +0200, Willy Tarreau wrote: > On Thu, Jun 13, 2019 at 05:31:09PM -0500, Dave Chiluk wrote: > > I used the number of calls to setsockopt with SO_LINGER in them using the > > following command. > > $ sudo timeout 60s strace -e setsockopt,close -p $(ps

Re: Increase in sockets in TIME_WAIT with 1.9.x

2019-06-13 Thread Willy Tarreau
On Thu, Jun 13, 2019 at 05:31:09PM -0500, Dave Chiluk wrote: > I was able to bisect this down to 53216e7 being the problematic commit, > when using calls to setsockopt(... SO_LINGER ...) as the test metric. Oh great, thank you for doing this! > I used the number of calls to setsockopt with

Re: Increase in sockets in TIME_WAIT with 1.9.x

2019-06-13 Thread Dave Chiluk
I was able to bisect this down to 53216e7 being the problematic commit, when using calls to setsockopt(... SO_LINGER ...) as the test metric. I used the number of calls to setsockopt with SO_LINGER in them using the following command. $ sudo timeout 60s strace -e setsockopt,close -p $(ps -lf -C

Re: Increase in sockets in TIME_WAIT with 1.9.x

2019-06-13 Thread Willy Tarreau
On Thu, Jun 13, 2019 at 03:20:20PM -0500, Dave Chiluk wrote: > I've attached an haproxy.cfg that is as minimal as I felt comfortable. (...) many thanks for this, Dave, I truly appreciate it. I'll have a look at it hopefully tomorrow morning. Willy

Re: Increase in sockets in TIME_WAIT with 1.9.x

2019-06-13 Thread Dave Chiluk
I've attached an haproxy.cfg that is as minimal as I felt comfortable. We are using admin sockets for dynamic configuration of backends so left the server-templating in, but no other application was configured to orchestrate haproxy at the time of testing. I've also attached output from $ sudo

Re: Increase in sockets in TIME_WAIT with 1.9.x

2019-06-13 Thread Willy Tarreau
On Wed, Jun 12, 2019 at 12:08:03PM -0500, Dave Chiluk wrote: > I did a bit more introspection on our TIME_WAIT connections. The increase > in sockets in TIME_WAIT is definitely from old connections to our backend > server instances. Considering the fact that this server is doesn't > actually

Re: Increase in sockets in TIME_WAIT with 1.9.x

2019-06-12 Thread Dave Chiluk
I did a bit more introspection on our TIME_WAIT connections. The increase in sockets in TIME_WAIT is definitely from old connections to our backend server instances. Considering the fact that this server is doesn't actually serve real traffic we can make a reasonable assumptions that this is

Re: Increase in sockets in TIME_WAIT with 1.9.x

2019-06-11 Thread Willy Tarreau
On Mon, Jun 10, 2019 at 04:01:27PM -0500, Dave Chiluk wrote: > We are in the process of evaluating upgrading to 1.9.8 from 1.8.17, > and we are seeing a roughly 70% increase in sockets in TIME_WAIT on > our haproxy servers with a mostly idle server cluster > $ sudo netstat | grep 'TIME_WAIT' | wc

Increase in sockets in TIME_WAIT with 1.9.x

2019-06-10 Thread Dave Chiluk
We are in the process of evaluating upgrading to 1.9.8 from 1.8.17, and we are seeing a roughly 70% increase in sockets in TIME_WAIT on our haproxy servers with a mostly idle server cluster $ sudo netstat | grep 'TIME_WAIT' | wc -l Looking at the source/destination of this it seems likely that