Hi Christopher,
Thanks for your reply.
Christopher Faulet schreef op 2022-08-04 08:56:
Le 8/3/22 à 16:23, William Edwards a écrit :
Hi,
Two days ago, I upgraded my first production system from HAProxy
1.8.19
to 2.2.9. Since then, many HTTP requests are hitting the server
timeout.
Before upgrade:
root@lb0-0:~# zgrep 'sD--' /var/log/haproxy.log.5.gz | wc -l
0
root@lb0-0:~# zgrep 'sD--' /var/log/haproxy.log.4.gz | wc -l
0
root@lb0-0:~# zgrep 'sD--' /var/log/haproxy.log.3.gz | wc -l
0
After upgrade:
# Day of upgrade
root@lb0-0:~# zgrep 'sD--' /var/log/haproxy.log.2.gz | wc -l
3798
# Yesterday
root@lb0-0:~# grep 'sD--' /var/log/haproxy.log.1 | wc -l
127176
# Today, so far
root@lb0-0:~# grep 'sD--' /var/log/haproxy.log | wc -l
85063
For this specific request, Ta ("total active time for the HTTP
request")
is 3, and Tt ("total TCP session duration time, between the moment the
proxy accepted it and the moment both ends were closed") is 300004 (5
minutes, the server timeout):
Aug 3 00:31:05 lb0-0 haproxy[16884]: $ip:62223
[03/Aug/2022:00:26:05.337] fr_other~
bk_http.lyr_http-lyr02.cf.ha.cyberfusion.cloud/http-lyr02.cf.ha.cyberfusion.cloud
0/0/0/3/300004 200 27992 - - sD-- 616/602/226/226/0 0/0 "GET
https://$domain/wp-content/uploads/2022/07/20220712_155022-300x300.jpg
HTTP/2.0"
The backend server indeed served the request within Ta:
$domain $ip - - [03/Aug/2022:00:26:05 +0200] "GET
/wp-content/uploads/2022/07/20220712_155022-300x300.jpg HTTP/1.1" 200
28008 "https://$domain/stoffen/"; "Mozilla/5.0 (Windows NT 10.0; Win64;
x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0
Safari/537.36"
The timeouts only occur with 5 out of 13 backends. There is no clear
pattern, i.e. the timeouts don't come in bursts, and they aren't
caused
by fixed clients.
Does anyone know why the TCP session is kept open, and the HTTP
request
is not responded to by HAProxy after the backend server responded to
the
HTTP request, but only after the server timeout is reached?
Hi,
The 2.2.9 is pretty old. [...] You must update it to
2.2.25 first.
The public outings regarding versioning contradict each other.
The bugs page says:
"If your version is not the last one in the maintenance branch, you are
missing fixes for known bugs, and by not updating you are needlessly
taking the responsibility for the risk of unexpected service outages and
exposing your web site to possible security issues."
However,
https://haproxy.debian.net/#distribution=Debian&release=buster&version=2.2
says:
"The Debian HAProxy packaging team provides various versions of HAProxy
packages for use on different Debian or Ubuntu systems. The following
wizard helps you to find the package suitable for your system. [...] You
will get a stable release of HAProxy 2.2: you may not get the latest
version but important fixes from later versions are included. Moreover,
regressions are unlikely."
The bugs page tries to get users to ALWAYS use the latest version. But
the haproxy.debian.org page says that it's okay not to use the latest
version.
It is affected by 369 known bugs
(http://www.haproxy.org/bugs/bugs-2.2.9.html).
I found this bug[1] on the bugs page which looks promising. I'll do some
more investigation today. Perhaps someone could corroborate that that
bug's symptoms match what I'm seeing.
Regards,
[1]: http://git.haproxy.org/?p=haproxy-2.2.git;a=commitdiff;h=3e2434e
--
With kind regards,
William Edwards
