Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

> On Apr 26, 2017, at Apr 26, 2:13 AM, jaseywang wrote: > > Hi > @Willy @Cyril do you have any recommended config for ssl related setting, we > now use nbproc and cpu-map to distribute the load to each cpu, though haproxy > can work with cdn now, it's performance is not

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

Hi @Willy @Cyril do you have any recommended config for ssl related setting, we now use nbproc and cpu-map to distribute the load to each cpu, though haproxy can work with cdn now, it's performance is not as good as before without cdn, user time of each core is almost saturated. Thanks. On Tue,

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

Awesome! It seems it's really something related with SSL handshakes, We open 80 port without https and let cdn connect to our 80 port, everything seems fine. I'll update this thread laters. On Tue, Apr 25, 2017 at 5:23 PM, Willy Tarreau wrote: > On Tue, Apr 25, 2017 at 05:00:48PM

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

On Tue, Apr 25, 2017 at 05:00:48PM +0800, jaseywang wrote: > Here is the data with debug mode off, still the same issue: > https://www.dropbox.com/s/4x0cjfv1o2kmwg3/analytics-debug-off.txt?dl=0 > > > Flat profile: > > Each sample counts as 0.01 seconds. > no time accumulated > > %

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

On Tue, Apr 25, 2017 at 03:32:16PM +0800, jaseywang wrote: > Here is the analytics data, dropbox link: > https://www.dropbox.com/s/rn92z42ao5l5rdo/analytics.txt?dl=0 Thank you. Well, everything there looks pretty good, with low CPU usage everywhere and normal number of calls for each function.

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

On Tue, Apr 25, 2017 at 02:26:53PM +0800, jaseywang wrote: > yes, really permission issue, here is the gmon.out file: > https://www.dropbox.com/s/c5wq7oxdjw0ddj0/gmon.out?dl=0 I can't do anything from it since it's specific to your executable. > not sure how this happen: > # gprof

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

yes, really permission issue, here is the gmon.out file: https://www.dropbox.com/s/c5wq7oxdjw0ddj0/gmon.out?dl=0 not sure how this happen: # gprof /usr/sbin/haproxy /var/lib/haproxy/gmon.out gprof: file `/usr/sbin/haproxy' has no symbols # gprof haproxy /var/lib/haproxy/gmon.out haproxy: No

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

On Tue, Apr 25, 2017 at 12:43:22PM +0800, jaseywang wrote: > Build options : > TARGET = linux2628 > CPU = generic > CC = gcc > CFLAGS = -pg -O2 -fwrapv -g -fno-strict-aliasing > -Wdeclaration-after-statement -DTCP_USER_TIMEOUT=18 > OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

Hi @Willy I compile the source with your make parameter like this: haproxy -vv HA-Proxy version 1.7.5 2017/04/03 Copyright 2000-2017 Willy Tarreau Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -pg -O2 -fwrapv -g -fno-strict-aliasing

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

Hi Cyril, On Mon, Apr 24, 2017 at 06:09:11PM +0200, Cyril Bonté wrote: > Hi Willy, > > > De: "Willy Tarreau" > > À: "jaseywang" > > [...] > > > > Below is the data during benchmark: > > > *maxsock = *136; *maxconn = *50; *maxpipes = *0 > > > current

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

Hi Willy, > De: "Willy Tarreau" > À: "jaseywang" [...] > > Below is the data during benchmark: > > *maxsock = *136; *maxconn = *50; *maxpipes = *0 > > current conns = 7488; current pipes = 0/0; conn rate = 322/sec > > Running tasks: 7366/7449; idle =

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

On Mon, Apr 24, 2017 at 08:33:14PM +0800, jaseywang wrote: > Before 08:24:30 PM, the data is normal, after that I began the benchmark, > here is data: > https://www.dropbox.com/s/yu2jsirv104ytwp/1.7.5-tcp-2?dl=0 > https://www.dropbox.com/s/8dap4kus5sj6by7/1.7.5-sess-2?dl=0 These ones also show

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

> > OK these ones are interesting. First, none of the CLOSE_WAIT connections > in the > netstat file is present in the "show sess" output. Might it be possible > that > these ones are left-overs from a previous run ? Could you please > double-check > that you don't have an older process still

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

On Mon, Apr 24, 2017 at 05:29:14PM +0800, jaseywang wrote: > Hi, > > Here is the 1.7.5 output with CDN, before 05:22:00 PM with timestamps in > the file, there is no request, since 05:22:00 PM, I began the benchmark, so > you can check from 05:22:00 PM. > 61.155.222.157 is cdn itself > > > The

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

Hi, Here is the 1.7.5 output with CDN, before 05:22:00 PM with timestamps in the file, there is no request, since 05:22:00 PM, I began the benchmark, so you can check from 05:22:00 PM. 61.155.222.157 is cdn itself The file is large here is the download link:

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

On Mon, Apr 24, 2017 at 02:43:54PM +0800, jaseywang wrote: > What you see is the data without cdn, you can get more data from the below > section: > > > Let haproxy sit behind in CDN, the session rate is around 270/s , current > > session is around 10k. below is the stats from haproxy and tcp.

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

What you see is the data without cdn, you can get more data from the below section: Let haproxy sit behind in CDN, the session rate is around 270/s , current > session is around 10k. below is the stats from haproxy and tcp. > current conns = 14269; current pipes = 0/0; conn rate = 270/sec >

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

Hi, On Fri, Apr 21, 2017 at 06:01:27PM +0800, jaseywang wrote: > Without CDN, the session rate is around 2k/s, current session is around 120: > current conns = 118; current pipes = 0/0; conn rate = 1938/sec > Running tasks: 2/127; idle = 42 % > > 111.111.111.111 is our haproxy public ip which

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

Really appreciate your time debugging this issue, it's really critical for our production env, if you need more details I'll try my best to provide them, thanks a lot! On Fri, Apr 21, 2017 at 10:48 PM, Willy Tarreau wrote: > On Fri, Apr 21, 2017 at 10:45:38PM +0800, jaseywang

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

On Fri, Apr 21, 2017 at 10:45:38PM +0800, jaseywang wrote: > Hi, @Willy, do you receive my updates, seems the mail message size is too > large? I got them but still no time to take a look, but yes I have them so no need to retransmit. Willy

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

Hi, @Willy, do you receive my updates, seems the mail message size is too large? On Fri, Apr 21, 2017 at 3:18 PM, Willy Tarreau wrote: > On Fri, Apr 21, 2017 at 02:37:52PM +0800, jaseywang wrote: > > > > > > Could you please confirm that most of the CLOSE_WAIT are on the front >

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

On Fri, Apr 21, 2017 at 02:37:52PM +0800, jaseywang wrote: > > > > Could you please confirm that most of the CLOSE_WAIT are on the front side > > and the ESTABLISHED on the backend side ? If that's the case, can you also > > please verify if there are pending data in the send queue for CLOSE_WAIT

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

> > Could you please confirm that most of the CLOSE_WAIT are on the front side > and the ESTABLISHED on the backend side ? If that's the case, can you also > please verify if there are pending data in the send queue for CLOSE_WAIT > sockets (3rd column in netstat) ? > Most of the

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

On Thu, Apr 20, 2017 at 11:03:42PM +0800, jaseywang wrote: > 1. The backlog of haproxy soon become full and begin to drop new tcp > connection since peak traffic begin, before CDN, our net.core.somaxconn is > 1024, and use default backlog of haproxy, everything performs well. After > using CDN,

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

On Fri, Apr 21, 2017 at 10:23:53AM +0800, jaseywang wrote: > Hi, Willy > Thanks for your help. We upgrade the version from 1.5.4 to 1.5.19, but > still the same issue, and what's your recommended version we can use for > production env? OK nice, at least you're not facing one of the many already

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

Hi, Willy Thanks for your help. We upgrade the version from 1.5.4 to 1.5.19, but still the same issue, and what's your recommended version we can use for production env? $ haproxy -vv HA-Proxy version 1.5.19 2016/12/25 Copyright 2000-2016 Willy Tarreau Build options :

Re: Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

Hello, On Thu, Apr 20, 2017 at 11:03:42PM +0800, jaseywang wrote: > Our haproxy 1.5.4 MS cluster performs quite well before, and the peak (...) > Now, the weird thing is why haproxy has so many closewait connections? and > why the backlog queue soon becomes full? Usually so many closewait means >

Haproxy 1.5.4 unable to accept new TCP request, backlog full, tens of thousands close_wait connection

Our haproxy 1.5.4 MS cluster performs quite well before, and the peak current connections is about 6k. Haproxy forward the request from client to Nginx, Nginx send the request to upstream JVM servers like this: client -> Haproxy -> Nginx -> upstream This week we use CDN to accept request from