On Fri, Apr 21, 2017 at 02:37:52PM +0800, jaseywang wrote:
> >
> > Could you please confirm that most of the CLOSE_WAIT are on the front side
> > and the ESTABLISHED on the backend side ? If that's the case, can you also
> > please verify if there are pending data in the send queue for CLOSE_WAIT
> > sockets (3rd column in netstat) ?
> >
> Most of the close_wait/established are on the haproxy side, between haproxy
> and cdn, not haproxy and nginx.
That makes sense for the close_wait but it's strange that the CDN establishes
tons of connections.
> what we see from netstat/ss is the recv queue is full, not the send-q,
So this means that incoming data are not consumed. Very strange. At least
it explains why they're in close_wait. The FIN has been received by the
system but not haproxy yet.
If the recv queue is full, it means these are POST requests (or at least
data uploads).
> below is our haproxy accept queue, which is 40000, when haproxy down, the
> recv-Q soon become full like 40000 or 40001:
> State Recv-Q Send-Q Local Address:Port Peer Address:Port
> LISTEN 0 40000 *:443 *:*
> users:(("haproxy",pid=152006,fd=8))
It can make sense if there's something preventing from processing these
connections correctly. You really need to look at your logs to see what
errors and/or termination flags are reported.
> I will give you more details after new benchmark to reproduce.
OK
Willy
