Hello,
since a while I see connection errors in my HAProxy-Logs, looking like this:
<134>Nov 20 13:19:10 haproxy[8]: <redacted>:60923 [20/Nov/2023:13:18:41.494]
http~ nextcloud/nextcloud 0/0/18/-1/28956 500 208 - - IH-- 19/19/0/0/0 0/0
{<redacted>} "PUT
https://<redacted>/remote.php/dav/uploads/<redacted>/5D56BCEB-AE7E-423A-B424-DCAB3F98C590/3
HTTP/2.0"
According to the documentation, a termination state of "I" should never happen
and be reported together with logs.
Now my Problem is: I don't have any more logs, besides that one line being
logged.
Therefore my question: What can I do to get further information about when/why
this occurs?
To give more background information: I am running HAProxy 2.9-dev10-db09cd6
(the docker image "haproxytech/haproxy-ubuntu-quic:2.9"
As you can see from the log line, (even though I have enabled HTTP/3), this
error occurs on a HTTP/2 SSL-Connection.The backend is a "NextCloud" instance.
So far, I only observed those "IH" Errors when uploading photos via the
smartphone application. I am not able to reproduce those errors on purpose, but
once one happens, there is a chance that retrying the request will produce the
same error again."In front" of nextcloud actually is an Apache2-Webserver (that
ships with the nextcloud docker container).
I was able to find the request in the apache logs:
<redacted> - <redacted> [20/Nov/2023:13:19:39 +0000] "PUT
/remote.php/dav/uploads/<redacted>/5D56BCEB-AE7E-423A-B424-DCAB3F98C590/3
HTTP/1.1" 204 656 "-" "Mozilla/5.0 (iOS) Nextcloud-iOS/4.9.1"
(Don't be surprised that the timestamps don't perfectly match. Apparently the
clock on both machines are not in sync).
(And: Actually this request returned a 204, because this was already a retry
form the client. In the initial request, the response was 201, but caused the
same IH error)
I tried restarting HAProxy multiple times, but every now and then, it happens
again.
I remember, that I also had that issue with older 2.9 builds, but I don't
remember any more, if this also happened on 2.8 build.
If helpful to you, this is my haproxy -vv:
HAProxy version 2.9-dev10-db09cd6 2023/11/18 - https://haproxy.org/Status:
development branch - not safe for use in production.Known bugs:
https://github.com/haproxy/haproxy/issues?q=is:issue+is:openRunning on: Linux
6.4.9-arch1-1 #1 SMP PREEMPT_DYNAMIC Tue, 08 Aug 2023 22:14:05 +0000
x86_64Build options : TARGET = linux-glibc CPU = generic CC = cc
CFLAGS = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement
-Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2
-Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member
-Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered
-Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int
-Wno-atomic-alignment OPTIONS = USE_PTHREAD_EMULATION=1 USE_LINUX_TPROXY=1
USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_TFO=1 USE_QUIC=1
USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1 DEBUG = -DDEBUG_STRICT
-DDEBUG_MEMORY_POOLS
Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H
-DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC
+LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING
+NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT
-PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX
+PTHREAD_EMULATION +QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL
-STATIC_PCRE -STATIC_PCRE2 -SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL
-ZLIB
Default settings : bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256,
default=8).Built with OpenSSL version : OpenSSL 3.1.2+quic 1 Aug 2023Running on
OpenSSL version : OpenSSL 3.1.2+quic 1 Aug 2023OpenSSL library supports TLS
extensions : yesOpenSSL library supports SNI : yesOpenSSL library supports :
TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3OpenSSL providers loaded : defaultBuilt with Lua
version : Lua 5.4.4Built with the Prometheus exporter as a serviceBuilt with
network namespace support.Built with libslz for stateless
compression.Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")Built with transparent
proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBINDBuilt with
PCRE2 version : 10.39 2021-10-29PCRE2 library supports JIT : yesEncrypted
password support via crypt(3): yesBuilt with gcc compiler version 11.4.0
Available polling systems : epoll : pref=300, test result OK poll :
pref=200, test result OK select : pref=150, test result OKTotal: 3 (3
usable), will use epoll.
Available multiplexer protocols :(protocols marked as <default> cannot be
specified using 'proto' keyword) quic : mode=HTTP side=FE mux=QUIC
flags=HTX|NO_UPG|FRAMED h2 : mode=HTTP side=FE|BE mux=H2
flags=HTX|HOL_RISK|NO_UPG fcgi : mode=HTTP side=BE mux=FCGI
flags=HTX|HOL_RISK|NO_UPG <default> : mode=HTTP side=FE|BE mux=H1
flags=HTX h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags= none : mode=TCP
side=FE|BE mux=PASS flags=NO_UPG
Available services : prometheus-exporterAvailable filters : [BWLIM] bwlim-in
[BWLIM] bwlim-out [CACHE] cache [COMP] compression [FCGI] fcgi-app [SPOE] spoe
[TRACE] trace
And finally the hopefully relevant bits of my config: (I left out unrelated
backends and "use_backend XX if {...}")
global stats socket /tmp/haproxy.sock mode 660 level admin expose-fd
listeners
maxconn 10000 log stdout local0 log stdout local1 notice
user haproxy stats timeout 30s daemon
ssl-default-bind-ciphers
ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
ssl-default-server-ciphers
ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
tune.ssl.default-dh-param 2048 tune.bufsize 10485760
peers myCluster peer <local-instance> *:10000 peer <peer-instance>
10.255.255.8:10000
defaults mode http option dontlognull timeout connect 5s timeout
client 10m timeout server 10m default-server inter 3s option
http-server-close errorfile 503 /usr/local/etc/haproxy/5xx.http
frontend http capture request header Host len 32 log-format %hr\ %r\ %st\
%B\ %Tr
bind *:80 bind *:443 ssl crt /usr/local/etc/haproxy/certs/ alpn
h2,http/1.1 bind quic4@:443 ssl crt /usr/local/etc/haproxy/certs/ alpn h3
mode http log global option httplog
maxconn 8000
stats enable stats uri /stats http-request use-service
prometheus-exporter if { path /<redacted>/prometheus/metrics } # sraped by
influxdb
http-response set-header Strict-Transport-Security max-age=16000000;\
includeSubDomains;\ preload; # 'Alt-Svc' header invites client to switch to
the QUIC protocol # Max age (ma) is set to 15 minutes (900 seconds), but #
can be increased once verified working as expected http-response set-header
alt-svc "h3=\":443\";ma=3600;"
acl internal_ip src 10.0.0.0/8 192.168.5.0/24 172.18.0.0/24
http-response set-header X-Frame-Options "SAMEORIGIN" http-response
set-header X-Content-Type-Options "nosniff"
http-request track-sc0 src table table-ip-whitelist if TRUE acl
isAuthenticated src_get_gpc0(table-ip-whitelist) gt 0 http-request
set-header X-Forwarded-For %[src] # Setting header explicit, as default
behavior is append
use_backend nextcloud if { hdr(Host) <redacted> } default_backend
deny
backend table-ip-whitelist stick-table type ip size 1m expire 86400s store
gpc0 peers myCluster
backend nextcloud mode http server nextcloud 192.168.5.63:80 check
is there a way to get more information about when/why the "IH" termination
happens?
Thanks a lotMarno
