Hi, I have a problem were I don't know were to look further and I hope I can get some pointers. The problem is that during times with a high request count sometimes answers for requests take a long time. So I identified an request that took a long time and got the log entries from the haproxy and the backend nginx server:
haproxy: Sep 26 09:47:25 heliosa haproxy[21758]: 109.41.0.201:18679 [26/Sep/2018:09:47:17.102] ext-www-shared-auction1-https~ www-shared-auction1-http/auction1-shared 0/0/0/5607/7748 200 663 - - ---- 6870/423/292/267/0 0/0 "GET /include/ajax/hotbid.php?PHPSESSID=qkmm012o23u8i9esj6nksfo445&_=1537945522572 HTTP/1.1" nginx (the logformat is non standard): "time": "2018-09-26T09:47:17+02:00" "body_bytes_sent": "237" "request_time": "0.036" "status": "200" "request": "GET /include/ajax/hotbid.php?PHPSESSID=qkmm012o23u8i9esj6nksfo445&_=1537945522572 HTTP/1.1" If I understand the haproxy default logformat correctly then the part 0/0/0/5607/7748 means that the request to the backend took around 7.748s (Tq, Tw and Tc are zero and Tr = 5607ms and Td = 2141ms; which means Tt = 7748ms is more or less completely used talking to the backend). But the backend server says it only used 0.036s to handle the request. I made a tcpdump on the server running the haproxy instance (the tcpdump of the request is attached) and if the timestamps are correct the request was indeed received in 0.039s. So either I made some mistake or it took around 5 seconds transfering/processing the request from the network interface into the haproxy instance. I looked into the statistics recorded of the server running haproxy and nothing seems noteworthy (no dropped packages or errors on the network interfaces, acceptable cpu and memory usage). The system is running haproxy 1.6.14 on an old Ubuntu trusty with 3.13.0-158 ubuntu kernel. It worked in the past without those problems. I am out of ideas where to look further for the cause of this issue and as said in the beginning any pointers or ideas were to look further would be greatly appreciated. Thanks Reinhard
request_qkmm012o23u8i9esj6nksfo445_1537945522572.pcap
Description: application/vnd.tcpdump.pcap
