On Mon, Apr 24, 2017 at 02:43:54PM +0800, jaseywang wrote: > What you see is the data without cdn, you can get more data from the below > section: > > > Let haproxy sit behind in CDN, the session rate is around 270/s , current > > session is around 10k. below is the stats from haproxy and tcp. > > current conns = 14269; current pipes = 0/0; conn rate = 270/sec > > Running tasks: 6136/6143; idle = 0 % > > Due to the tcp and haproxy stats is too large, I upload the monitoring > > data to dropbox: > > https://www.dropbox.com/s/zdyqn4ohvzv47zb/lb-sess?dl=0 > > https://www.dropbox.com/s/qrs7vzbcm8m2kwk/lb-tcp?dl=0
Ah OK, it was not easy to spot after 5500 lines of session dumps. I could download these two files. The first thing I'm noticing is that the few affected sessions I checked were all carrying HEAD requests and that the server takes a long time to respond to these requests (5-8s), maybe as a side effect of the large number of connections. I'm seeing a strange one here, the end-to-end connection is : 61.155.222.157:39891 --> [:443 -- :59323] --> 10.32.132.114:80 The connection arrived at 17:30:20, and presented a 854 bytes-long request, forwarded to the server over socket fd 5721. The response is never received, but the netstat output indicates that it was received. The server-side connection flags report that it's not polling for reads, so that explains it. From this point I cannot dig further, because this situation might be caused by one of the 183 bugs already fixed and complicates the diagnostic. We really need to have a dump produced using an up-to-date version. Since you said you tried 1.7.5 and you got the same problem using it, please run the captures using this version, at least it is not affected by all the problems we already fixed over the last 2.5 years. Regards, willy
