Hello,
This is my 1st cry for help on HAProxy here. If this is not the
correct place, please be so kind as to redirect me to the proper one.
I'm new to HAProxy, and I'm trying to set up HAProxy 2.5.0, to act as
an SSL terminator to a single backend, for my initial testing.
The connection itself, as well the communication between the client,
all the way to the backend is operational. The issue I'm having is
with http keepalive. The description is as follows:
- http keepalive is operating properly between client and HAProxy,
respecting all parameters provided/defined on both client and HAProxy
- http keepalive is also operating as expected between HAProxy and the
backend server, as long as there's less than 10 seconds between
transfers from the client
if more than 10 seconds pass between any communication is received
from the client side, this is what happens, at second T+10:
- There is no extra data produced by the client, nothing on tcpdump
- Between HAProxy and the backend, this happens:
17:22:54.058275 IP 10.116.0.96.443 > 10.116.0.65.36096: Flags [P.],
seq 3889:3920, ack 832, win 505, options [nop,nop,TS val 2833024558
ecr 864209762], length 31
17:22:54.058330 IP 10.116.0.65.36096 > 10.116.0.96.443: Flags [.], ack
3920, win 501, options [nop,nop,TS val 864219772 ecr 2833024558],
length 0
17:22:54.058366 IP 10.116.0.96.443 > 10.116.0.65.36096: Flags [F.],
seq 3920, ack 832, win 505, options [nop,nop,TS val 2833024558 ecr
864219772], length 0
17:22:54.058500 IP 10.116.0.65.36096 > 10.116.0.96.443: Flags [P.],
seq 832:863, ack 3921, win 501, options [nop,nop,TS val 864219772 ecr
2833024558], length 31
17:22:54.058516 IP 10.116.0.96.443 > 10.116.0.65.36096: Flags [R], seq
443460723, win 0, length 0
which culminates is the session being closed. This capture was taken
on the backend server.
This always happens after 10 seconds, which led me to believe it's a
timeout on the HAProxy side, but I was unable to find any parameter to
adjust it, when looking at the documentation.
As an extra note, if I communicate from the client, towards the
backend directly, http keepalive also works as it should.
Here the HAProxy configuration file:
global
log /dev/log local0 info
defaults
mode tcp
timeout connect 500s
timeout client 500s
timeout server 500s
maxconn 300000
timeout http-request 500s
timeout http-keep-alive 500s
frontend all-in
mode http
bind 1.1.1.1:443 ssl crt /etc/haproxy/ssl_certs/somedomain.pem
tcp-request inspect-delay 5s
use_backend somedomain if { ssl_fc_sni_end .somedomain.com }
option forwardfor
timeout client 2147483647
backend somedomain
mode http
balance source
hash-type consistent
http-reuse always
server somedomain 1.1.1.1:8443 ssl verify none alpn http/1.1
no option http-server-close
no option httpclose
option forwardfor
timeout http-request 500s
timeout http-keep-alive 500s
timeout server 2147483647
The public IPs were hidden for privacy.
The setup is one physical host that holds HAProxy, and the backend is
on a docker container, on the same host.
Also some HAProxy information:
# haproxy -vv
HAProxy version 2.5.0 2021/11/23 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2023.
Known bugs: http://www.haproxy.org/bugs/bugs-2.5.0.html
Running on: Linux 5.9.11-1.el7.elrepo.x86_64 #1 SMP Tue Nov 24
09:45:34 EST 2020 x86_64
Build options :
TARGET = linux-glibc
CPU = generic
CC = cc
CFLAGS = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement
-fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter
-Wno-clobbered -Wno-missing-field-initializers -Wtype-limits
-Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond
-Wnull-dereference
OPTIONS = USE_PCRE2_JIT=1 USE_THREAD=1 USE_LIBCRYPT=1 USE_OPENSSL=1
USE_LUA=1 USE_SLZ=1 USE_SYSTEMD=1
DEBUG =
Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT -PCRE2
+PCRE2_JIT +POLL +THREAD +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY
+LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H +GETADDRINFO +OPENSSL
+LUA +ACCEPT4 -CLOSEFROM -ZLIB +SLZ +CPU_AFFINITY +TFO +NS +DL +RT
-DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL
-PROCCTL +THREAD_DUMP -EVPORTS -OT -QUIC -PROMEX -MEMORY_PROFILING
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=48).
Built with OpenSSL version : OpenSSL 1.1.1k FIPS 25 Mar 2021
Running on OpenSSL version : OpenSSL 1.1.1k FIPS 25 Mar 2021
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.4.3
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.23 2017-02-14
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 7.3.1 20180303 (Red Hat 7.3.1-5)
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTTP side=FE|BE mux=H2
flags=HTX|CLEAN_ABRT|HOL_RISK|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI
flags=HTX|HOL_RISK|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
Available services : none
Available filters :
[SPOE] spoe
[CACHE] cache
[FCGI] fcgi-app
[COMP] compression
[TRACE] trace
Can someone help me, on why I'm unable to get http keepalive working
with idle times longer than 10 seconds, from the HAProxy instance to
the backend?
Thanks in advance for any help given :)
Regards,
--
Rui Santos
Veni, Vidi, Linux
